[ { "path": ".gitignore", "content": "# Intermediate and built stuff.\n.sass-cache/\n/build/\n/gen/\nclox\n*.class\nexercises/chapter01_introduction/3/linked_list\n.idea/\n\n# I keep a scratch file at the top level to try stuff out.\ntemp.lox\n\n# XCode user-specific stuff.\nxcuserdata/\n\n# Dart stuff.\n/tool/.dart_tool/\n/tool/.packages\n" }, { "path": "LICENSE", "content": "Copyright (c) 2015 Robert Nystrom\n\n---------------------------------- Commentary ----------------------------------\n\nThe licensing story for this repository is a little complex. Here's my\nmotivation:\n\n* I want you to get as much use out of the material here as possible. I wrote\n this book to help you, and I don't want you to be encumbered when it comes to\n making the most of it. That's also why I put it online for free.\n\n* With my previous book, collaboration on GitHub was immesensely helpful. I want\n to ensure people can fork the repo, send me fixes, etc. without violating the\n license or feeling weird.\n\n* When it comes to code, I'm completely comfortable with people redistributing,\n remixing, changing, whatever with it. I've been using the MIT license for open\n source stuff for decades.\n\n This book contains two complete interpreters and I would be delighted for them\n to be the jumping-off point for any number of real full-featured language\n implementations.\n\n* When it comes to my prose, illustrations, and the visual design of the site,\n that feels a little more, I don't know, *me* than the code. The words are in\n my voice, the drawings are literally my handwriting, and the look of the site\n is part of the book's and, by extension, my brand.\n\n I feel weird thinking about someone, say taking one of the chapters and making\n significant changes to it to fit their writing style while still having some\n of it read like it came from me. Likewise, I'd be sad to see another site\n online that looked exactly like mine because it reuses my stylesheets.\n\n* My previous book ended up being translated into several languages. I want to\n be careful to not be so permissive that it prevents me from signing typical\n contracts that give them exclusive translation rights to certain territories\n and languages.\n\n* If I allow the prose and illustrations to be redistributed commercially, there\n is nothing preventing someone from slapping together a cheap print or ebook\n version of the book and putting it up for sale. I'm not too worried about my\n own sales being undercut, but I very much want to avoid readers finding\n themselves with a low quality book that they incorrectly think is from me.\n\n I worked very hard on this book. I want you to get the best possible\n experience.\n\nAll of this is way more complex than I'd like, especially since my brain isn't\nwired to care about intellectual property. I like thinking about making stuff,\nnot thinking about the legal rights around the stuff I made. (If your brain is\nwired to think about legal stuff and you see that I'm doing something dumb,\nplease do let me know.)\n\nThe best solution I've been able to come up with is to use two licenses:\n\n---------------------------------- License(s) ----------------------------------\n\nEach file in this repository falls under one of two licenses. Files whose\nextension is \".c\", \".dart\", \".h\", \".java\", or \".lox\" use the MIT license:\n\n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to\n deal in the Software without restriction, including without limitation the\n rights to use, copy, modify, merge, publish, distribute, sublicense, and/or\n sell copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n\n The above copyright notice and this permission notice shall be included in\n all copies or substantial portions of the Software.\n\n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS\n IN THE SOFTWARE.\n\nAll other files, including (but not limited to) \".md\" (except for\n\"book/appendix-i.md\" which uses the MIT license above), \".png\", \".jpg\", \".html\",\n\".scss\", \".css\", and \".txt\" use this Creative Commons license:\n\n Attribution-NonCommercial-NoDerivatives 4.0\n International (CC BY-NC-ND 4.0)\n\n https://creativecommons.org/licenses/by-nc-nd/4.0/\n" }, { "path": "Makefile", "content": "BUILD_DIR := build\nTOOL_SOURCES := tool/pubspec.lock $(shell find tool -name '*.dart')\nBUILD_SNAPSHOT := $(BUILD_DIR)/build.dart.snapshot\nTEST_SNAPSHOT := $(BUILD_DIR)/test.dart.snapshot\n\ndefault: book clox jlox\n\n# Run dart pub get on tool directory.\nget:\n\t@ cd ./tool; dart pub get\n\n# Remove all build outputs and intermediate files.\nclean:\n\t@ rm -rf $(BUILD_DIR)\n\t@ rm -rf gen\n\n# Build the site.\nbook: $(BUILD_SNAPSHOT)\n\t@ dart $(BUILD_SNAPSHOT)\n\n# Run a local development server for the site that rebuilds automatically.\nserve: $(BUILD_SNAPSHOT)\n\t@ dart $(BUILD_SNAPSHOT) --serve\n\n$(BUILD_SNAPSHOT): $(TOOL_SOURCES)\n\t@ mkdir -p build\n\t@ echo \"Compiling Dart snapshot...\"\n\t@ dart --snapshot=$@ --snapshot-kind=app-jit tool/bin/build.dart >/dev/null\n\n# Run the tests for the final versions of clox and jlox.\ntest: debug jlox $(TEST_SNAPSHOT)\n\t@- dart $(TEST_SNAPSHOT) clox\n\t@ dart $(TEST_SNAPSHOT) jlox\n\n# Run the tests for the final version of clox.\ntest_clox: debug $(TEST_SNAPSHOT)\n\t@ dart $(TEST_SNAPSHOT) clox\n\n# Run the tests for the final version of jlox.\ntest_jlox: jlox $(TEST_SNAPSHOT)\n\t@ dart $(TEST_SNAPSHOT) jlox\n\n# Run the tests for every chapter's version of clox.\ntest_c: debug c_chapters $(TEST_SNAPSHOT)\n\t@ dart $(TEST_SNAPSHOT) c\n\n# Run the tests for every chapter's version of jlox.\ntest_java: jlox java_chapters $(TEST_SNAPSHOT)\n\t@ dart $(TEST_SNAPSHOT) java\n\n# Run the tests for every chapter's version of clox and jlox.\ntest_all: debug jlox c_chapters java_chapters compile_snippets $(TEST_SNAPSHOT)\n\t@ dart $(TEST_SNAPSHOT) all\n\n$(TEST_SNAPSHOT): $(TOOL_SOURCES)\n\t@ mkdir -p build\n\t@ echo \"Compiling Dart snapshot...\"\n\t@ dart --snapshot=$@ --snapshot-kind=app-jit tool/bin/test.dart clox >/dev/null\n\n# Compile a debug build of clox.\ndebug:\n\t@ $(MAKE) -f util/c.make NAME=cloxd MODE=debug SOURCE_DIR=c\n\n# Compile the C interpreter.\nclox:\n\t@ $(MAKE) -f util/c.make NAME=clox MODE=release SOURCE_DIR=c\n\t@ cp build/clox clox # For convenience, copy the interpreter to the top level.\n\n# Compile the C interpreter as ANSI standard C++.\ncpplox:\n\t@ $(MAKE) -f util/c.make NAME=cpplox MODE=debug CPP=true SOURCE_DIR=c\n\n# Compile and run the AST generator.\ngenerate_ast:\n\t@ $(MAKE) -f util/java.make DIR=java PACKAGE=tool\n\t@ java -cp build/java com.craftinginterpreters.tool.GenerateAst \\\n\t\t\tjava/com/craftinginterpreters/lox\n\n# Compile the Java interpreter .java files to .class files.\njlox: generate_ast\n\t@ $(MAKE) -f util/java.make DIR=java PACKAGE=lox\n\nrun_generate_ast = @ java -cp build/gen/$(1) \\\n\t\t\tcom.craftinginterpreters.tool.GenerateAst \\\n\t\t\tgen/$(1)/com/craftinginterpreters/lox\n\njava_chapters: split_chapters\n\t@ $(MAKE) -f util/java.make DIR=gen/chap04_scanning PACKAGE=lox\n\n\t@ $(MAKE) -f util/java.make DIR=gen/chap05_representing PACKAGE=tool\n\t$(call run_generate_ast,chap05_representing)\n\t@ $(MAKE) -f util/java.make DIR=gen/chap05_representing PACKAGE=lox\n\n\t@ $(MAKE) -f util/java.make DIR=gen/chap06_parsing PACKAGE=tool\n\t$(call run_generate_ast,chap06_parsing)\n\t@ $(MAKE) -f util/java.make DIR=gen/chap06_parsing PACKAGE=lox\n\n\t@ $(MAKE) -f util/java.make DIR=gen/chap07_evaluating PACKAGE=tool\n\t$(call run_generate_ast,chap07_evaluating)\n\t@ $(MAKE) -f util/java.make DIR=gen/chap07_evaluating PACKAGE=lox\n\n\t@ $(MAKE) -f util/java.make DIR=gen/chap08_statements PACKAGE=tool\n\t$(call run_generate_ast,chap08_statements)\n\t@ $(MAKE) -f util/java.make DIR=gen/chap08_statements PACKAGE=lox\n\n\t@ $(MAKE) -f util/java.make DIR=gen/chap09_control PACKAGE=tool\n\t$(call run_generate_ast,chap09_control)\n\t@ $(MAKE) -f util/java.make DIR=gen/chap09_control PACKAGE=lox\n\n\t@ $(MAKE) -f util/java.make DIR=gen/chap10_functions PACKAGE=tool\n\t$(call run_generate_ast,chap10_functions)\n\t@ $(MAKE) -f util/java.make DIR=gen/chap10_functions PACKAGE=lox\n\n\t@ $(MAKE) -f util/java.make DIR=gen/chap11_resolving PACKAGE=tool\n\t$(call run_generate_ast,chap11_resolving)\n\t@ $(MAKE) -f util/java.make DIR=gen/chap11_resolving PACKAGE=lox\n\n\t@ $(MAKE) -f util/java.make DIR=gen/chap12_classes PACKAGE=tool\n\t$(call run_generate_ast,chap12_classes)\n\t@ $(MAKE) -f util/java.make DIR=gen/chap12_classes PACKAGE=lox\n\n\t@ $(MAKE) -f util/java.make DIR=gen/chap13_inheritance PACKAGE=tool\n\t$(call run_generate_ast,chap13_inheritance)\n\t@ $(MAKE) -f util/java.make DIR=gen/chap13_inheritance PACKAGE=lox\n\nc_chapters: split_chapters\n\t@ $(MAKE) -f util/c.make NAME=chap14_chunks MODE=release SOURCE_DIR=gen/chap14_chunks\n\t@ $(MAKE) -f util/c.make NAME=chap15_virtual MODE=release SOURCE_DIR=gen/chap15_virtual\n\t@ $(MAKE) -f util/c.make NAME=chap16_scanning MODE=release SOURCE_DIR=gen/chap16_scanning\n\t@ $(MAKE) -f util/c.make NAME=chap17_compiling MODE=release SOURCE_DIR=gen/chap17_compiling\n\t@ $(MAKE) -f util/c.make NAME=chap18_types MODE=release SOURCE_DIR=gen/chap18_types\n\t@ $(MAKE) -f util/c.make NAME=chap19_strings MODE=release SOURCE_DIR=gen/chap19_strings\n\t@ $(MAKE) -f util/c.make NAME=chap20_hash MODE=release SOURCE_DIR=gen/chap20_hash\n\t@ $(MAKE) -f util/c.make NAME=chap21_global MODE=release SOURCE_DIR=gen/chap21_global\n\t@ $(MAKE) -f util/c.make NAME=chap22_local MODE=release SOURCE_DIR=gen/chap22_local\n\t@ $(MAKE) -f util/c.make NAME=chap23_jumping MODE=release SOURCE_DIR=gen/chap23_jumping\n\t@ $(MAKE) -f util/c.make NAME=chap24_calls MODE=release SOURCE_DIR=gen/chap24_calls\n\t@ $(MAKE) -f util/c.make NAME=chap25_closures MODE=release SOURCE_DIR=gen/chap25_closures\n\t@ $(MAKE) -f util/c.make NAME=chap26_garbage MODE=release SOURCE_DIR=gen/chap26_garbage\n\t@ $(MAKE) -f util/c.make NAME=chap27_classes MODE=release SOURCE_DIR=gen/chap27_classes\n\t@ $(MAKE) -f util/c.make NAME=chap28_methods MODE=release SOURCE_DIR=gen/chap28_methods\n\t@ $(MAKE) -f util/c.make NAME=chap29_superclasses MODE=release SOURCE_DIR=gen/chap29_superclasses\n\t@ $(MAKE) -f util/c.make NAME=chap30_optimization MODE=release SOURCE_DIR=gen/chap30_optimization\n\ncpp_chapters: split_chapters\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap14_chunks MODE=release CPP=true SOURCE_DIR=gen/chap14_chunks\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap15_virtual MODE=release CPP=true SOURCE_DIR=gen/chap15_virtual\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap16_scanning MODE=release CPP=true SOURCE_DIR=gen/chap16_scanning\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap17_compiling MODE=release CPP=true SOURCE_DIR=gen/chap17_compiling\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap18_types MODE=release CPP=true SOURCE_DIR=gen/chap18_types\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap19_strings MODE=release CPP=true SOURCE_DIR=gen/chap19_strings\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap20_hash MODE=release CPP=true SOURCE_DIR=gen/chap20_hash\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap21_global MODE=release CPP=true SOURCE_DIR=gen/chap21_global\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap22_local MODE=release CPP=true SOURCE_DIR=gen/chap22_local\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap23_jumping MODE=release CPP=true SOURCE_DIR=gen/chap23_jumping\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap24_calls MODE=release CPP=true SOURCE_DIR=gen/chap24_calls\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap25_closures MODE=release CPP=true SOURCE_DIR=gen/chap25_closures\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap26_garbage MODE=release CPP=true SOURCE_DIR=gen/chap26_garbage\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap27_classes MODE=release CPP=true SOURCE_DIR=gen/chap27_classes\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap28_methods MODE=release CPP=true SOURCE_DIR=gen/chap28_methods\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap29_superclasses MODE=release CPP=true SOURCE_DIR=gen/chap29_superclasses\n\t@ $(MAKE) -f util/c.make NAME=cpp_chap30_optimization MODE=release CPP=true SOURCE_DIR=gen/chap30_optimization\n\ndiffs: split_chapters java_chapters\n\t@ mkdir -p build/diffs\n\t@ -diff --recursive --new-file nonexistent/ gen/chap04_scanning/com/craftinginterpreters/ > build/diffs/chap04_scanning.diff\n\t@ -diff --recursive --new-file gen/chap04_scanning/com/craftinginterpreters/ gen/chap05_representing/com/craftinginterpreters/ > build/diffs/chap05_representing.diff\n\t@ -diff --recursive --new-file gen/chap05_representing/com/craftinginterpreters/ gen/chap06_parsing/com/craftinginterpreters/ > build/diffs/chap06_parsing.diff\n\t@ -diff --recursive --new-file gen/chap06_parsing/com/craftinginterpreters/ gen/chap07_evaluating/com/craftinginterpreters/ > build/diffs/chap07_evaluating.diff\n\t@ -diff --recursive --new-file gen/chap07_evaluating/com/craftinginterpreters/ gen/chap08_statements/com/craftinginterpreters/ > build/diffs/chap08_statements.diff\n\t@ -diff --recursive --new-file gen/chap08_statements/com/craftinginterpreters/ gen/chap09_control/com/craftinginterpreters/ > build/diffs/chap09_control.diff\n\t@ -diff --recursive --new-file gen/chap09_control/com/craftinginterpreters/ gen/chap10_functions/com/craftinginterpreters/ > build/diffs/chap10_functions.diff\n\t@ -diff --recursive --new-file gen/chap10_functions/com/craftinginterpreters/ gen/chap11_resolving/com/craftinginterpreters/ > build/diffs/chap11_resolving.diff\n\t@ -diff --recursive --new-file gen/chap11_resolving/com/craftinginterpreters/ gen/chap12_classes/com/craftinginterpreters/ > build/diffs/chap12_classes.diff\n\t@ -diff --recursive --new-file gen/chap12_classes/com/craftinginterpreters/ gen/chap13_inheritance/com/craftinginterpreters/ > build/diffs/chap13_inheritance.diff\n\n\t@ -diff --new-file nonexistent/ gen/chap14_chunks/ > build/diffs/chap14_chunks.diff\n\t@ -diff --new-file gen/chap14_chunks/ gen/chap15_virtual/ > build/diffs/chap15_virtual.diff\n\t@ -diff --new-file gen/chap15_virtual/ gen/chap16_scanning/ > build/diffs/chap16_scanning.diff\n\t@ -diff --new-file gen/chap16_scanning/ gen/chap17_compiling/ > build/diffs/chap17_compiling.diff\n\t@ -diff --new-file gen/chap17_compiling/ gen/chap18_types/ > build/diffs/chap18_types.diff\n\t@ -diff --new-file gen/chap18_types/ gen/chap19_strings/ > build/diffs/chap19_strings.diff\n\t@ -diff --new-file gen/chap19_strings/ gen/chap20_hash/ > build/diffs/chap20_hash.diff\n\t@ -diff --new-file gen/chap20_hash/ gen/chap21_global/ > build/diffs/chap21_global.diff\n\t@ -diff --new-file gen/chap21_global/ gen/chap22_local/ > build/diffs/chap22_local.diff\n\t@ -diff --new-file gen/chap22_local/ gen/chap23_jumping/ > build/diffs/chap23_jumping.diff\n\t@ -diff --new-file gen/chap23_jumping/ gen/chap24_calls/ > build/diffs/chap24_calls.diff\n\t@ -diff --new-file gen/chap24_calls/ gen/chap25_closures/ > build/diffs/chap25_closures.diff\n\t@ -diff --new-file gen/chap25_closures/ gen/chap26_garbage/ > build/diffs/chap26_garbage.diff\n\t@ -diff --new-file gen/chap26_garbage/ gen/chap27_classes/ > build/diffs/chap27_classes.diff\n\t@ -diff --new-file gen/chap27_classes/ gen/chap28_methods/ > build/diffs/chap28_methods.diff\n\t@ -diff --new-file gen/chap28_methods/ gen/chap29_superclasses/ > build/diffs/chap29_superclasses.diff\n\t@ -diff --new-file gen/chap29_superclasses/ gen/chap30_optimization/ > build/diffs/chap30_optimization.diff\n\nsplit_chapters:\n\t@ dart tool/bin/split_chapters.dart\n\ncompile_snippets:\n\t@ dart tool/bin/compile_snippets.dart\n\n# Generate the XML for importing into InDesign.\nxml: $(TOOL_SOURCES)\n\t@ dart --enable-asserts tool/bin/build_xml.dart\n\n.PHONY: book c_chapters clean clox compile_snippets debug default diffs \\\n\tget java_chapters jlox serve split_chapters test test_all test_c test_java\n" }, { "path": "README.md", "content": "This is the repo used for the in-progress book \"[Crafting Interpreters][]\". It\ncontains the Markdown text of the book, full implementations of both\ninterpreters, as well as the build system to weave the two together into the\nfinal site.\n\n[crafting interpreters]: http://craftinginterpreters.com\n\nIf you find an error or have a suggestion, please do file an issue here. Thank\nyou!\n\n## Contributing\n\nOne of the absolute best things about writing a book online and putting it out\nthere before it's done is that people like you have been kind enough to give me\nfeedback, point out typos, and find other errors or unclear text.\n\nIf you'd like to do that, great! You can just file bugs here on the repo, or\nsend a pull request if you're so inclined. If you want to send a pull request,\nbut don't want to get the build system set up to regenerate the HTML too, don't\nworry about it. I'll do that when I pull it in.\n\n## Ports and implementations\n\nAnother way to get involved is by sharing your own implementation of Lox. Ports\nto other languages are particularly useful since not every reader likes Java and\nC. Feel free to add your Lox port or implementation to the wiki:\n\n* [Lox implementations][]\n\n[lox implementations]: https://github.com/munificent/craftinginterpreters/wiki/Lox-implementations\n\n## Building Stuff\n\nI am a terribly forgetful, error-prone mammal, so I automated as much as I\ncould.\n\n### Prerequisites\n\nI develop on an OS X machine, but any POSIX system should work too. With a\nlittle extra effort, you should be able to get this working on Windows as well,\nthough I can't help you out much.\n\nMost of the work is orchestrated by make. The build scripts, test runner, and\nother utilities are all written in [Dart][]. Instructions to install Dart are\n[here][install]. Once you have Dart installed and on your path, run:\n\n```sh\n$ make get\n```\n\n[dart]: https://dart.dev/\n[install]: https://dart.dev/get-dart\n\nThis downloads all of the packages used by the build and test scripts.\n\nIn order to compile the two interpreters, you also need a C compiler on your\npath as well as `javac`.\n\n### Building\n\nOnce you've got that setup, try:\n\n```sh\n$ make\n```\n\nIf everything is working, that will generate the site for the book as well as\ncompiling the two interpreters clox and jlox. You can run either interpreter\nright from the root of the repo:\n\n```sh\n$ ./clox\n$ ./jlox\n```\n\n### Hacking on the book\n\nThe Markdown and snippets of source code are woven together into the final HTML\nusing a hand-written static site generator that started out as a [single tiny\nPython script][py] for [my first book][gpp] and somehow grew into something\napproximating a real program.\n\n[py]: https://github.com/munificent/game-programming-patterns/blob/master/script/format.py\n[gpp]: http://gameprogrammingpatterns.com/\n\nThe generated HTML is committed in the repo under `site/`. It is built from a\ncombination of Markdown for prose, which lives in `book/`, and snippets of code\nthat are weaved in from the Java and C implementations in `java/` and `c/`. (All\nof those funny looking comments in the source code are how it knows which\nsnippet goes where.)\n\nThe script that does all the magic is `tool/bin/build.dart`. You can run that\ndirectly, or run:\n\n```sh\n$ make book\n```\n\nThat generates the entire site in one batch. If you are incrementally working\non it, you'll want to run the development server:\n\n```sh\n$ make serve\n```\n\nThis runs a little HTTP server on localhost rooted at the `site/` directory.\nAny time you request a page, it regenerates any files whose sources have been\nchanged, including Markdown files, interpreter source files, templates, and\nassets. Just let that keep running, edit files locally, and refresh your\nbrowser to see the changes.\n\n### Building the interpreters\n\nYou can build each interpreter like so:\n\n```sh\n$ make clox\n$ make jlox\n```\n\nThis builds the final version of each interpreter as it appears at the end of\nits part in the book.\n\nYou can also see what the interpreters look like at the end of each chapter. (I\nuse this to make sure they are working even in the middle of the book.) This is\ndriven by a script, `tool/bin/split_chapters.dart` that uses the same comment\nmarkers for the code snippets to determine which chunks of code are present in\neach chapter. It takes only the snippets that have been seen by the end of each\nchapter and produces a new copy of the source in `gen/`, one directory for each\nchapter's code. (These are also an easier way to view the source code since they\nhave all of the distracting marker comments stripped out.)\n\nThen, each of those can be built separately. Run:\n\n```sh\n$ make c_chapters\n```\n\nAnd in the `build/` directory, you'll get an executable for each chapter, like\n`chap14_chunks`, etc. Likewise:\n\n```sh\n$ make java_chapters\n```\n\nThis compiles the Java code to classfiles in `build/gen/` in a subdirectory for\neach chapter.\n\n## Testing\n\nI have a full Lox test suite that I use to ensure the interpreters in the book\ndo what they're supposed to do. The test cases live in `test/`. The Dart\nprogram `tool/bin/test.dart` is a test runner that runs each of those test\nfiles on a Lox interpreter, parses the result, and validates that that the test\ndoes what it's expected to do.\n\nThere are various interpreters you can run the tests against:\n\n```sh\n$ make test # The final versions of clox and jlox.\n$ make test_clox # The final version of clox.\n$ make test_jlox # The final version of jlox.\n$ make test_c # Every chapter's version of clox.\n$ make test_java # Every chapter's version of jlox.\n$ make test_all # All of the above.\n```\n\n### Testing your implementation\n\nYou are welcome to use the test suite and the test runner to test your own Lox\nimplementation. The test runner is at `tool/bin/test.dart` and can be given a\ncustom interpreter executable to run using `--interpreter`. For example, if you\nhad an interpreter executable at `my_code/boblox`, you could test it like:\n\n```sh\n$ dart tool/bin/test.dart clox --interpreter my_code/boblox\n```\n\nYou still need to tell it which suite of tests to run because that determines\nthe test expectations. If your interpreter should behave like jlox, use \"jlox\"\nas the suite name. If it behaves like clox, use \"clox\". If your interpreter is\nonly complete up to the end of one of the chapters in the book, you can use\nthat chapter as the suite, like \"chap10_functions\". See the Makefile for the\nnames of all of the chapters.\n\nIf your interpreter needs other command line arguments passed to use, pass them\nto the test runner using `--arguments` and it will forward to your interpreter.\n\n## Repository Layout\n\n* `asset/` – Sass files and jinja2 templates used to generate the site.\n* `book/` - Markdown files for the text of each chapter.\n* `build/` - Intermediate files and other build output (except for the site\n itself) go here. Not committed to Git.\n* `c/` – Source code of clox, the interpreter written in C. Also contains an\n XCode project, if that's your thing.\n* `gen/` – Java source files generated by GenerateAst.java go here. Not\n committed.\n* `java/` – Source code of jlox, the interpreter written in Java.\n* `note/` – Various research, notes, TODOs, and other miscellanea.\n* `note/answers` – Sample answers for the challenges. No cheating!\n* `site/` – The final generated site. The contents of this directory directly\n mirror craftinginterpreters.com. Most content here is generated by build.py,\n but fonts, images, and JS only live here. Everything is committed, even the\n generated content.\n* `test/` – Test cases for the Lox implementations.\n* `tool/` – Dart package containing the build, test, and other scripts.\n" }, { "path": "asset/index.scss", "content": "@import 'sass/shared';\n@import 'sass/sign-up';\n\nbody, h1, h2, h3, h4, p, blockquote, code, ul, ol, dl, dd, img {\n margin: 0;\n}\n\nbody {\n background: $dark url('image/background.png') top center / 100% auto no-repeat;\n color: #222;\n font: normal 16px/24px $serif;\n}\n\na {\n color: $primary;\n text-decoration: none;\n\n border-bottom: solid 1px transparentize($light, 1.0);\n\n transition: color 0.2s ease,\n border-color 0.4s ease;\n}\n\na:hover {\n color: $primary;\n border-bottom: solid 1px opacify($light, 1.0);\n}\n\narticle {\n margin: 0 auto;\n padding: 0 0 12px 0;\n max-width: $col * 20;\n background: #fff;\n}\n\nheader {\n margin: 0 0 $col 0;\n color: $warm-dark;\n background: $warm-5;\n border-bottom: solid 1px $warm-4;\n}\n\nmain {\n margin: 0 $col;\n}\n\nimg.header {\n display: block;\n width: 100%;\n}\n\nimg.small {\n display: none;\n}\n\ndiv.intro {\n display: flex;\n\n blockquote {\n flex-basis: 40%;\n margin: 0 $col 0 0;\n font: italic 28px/42px $serif;\n }\n\n div.text {\n flex-basis: 60%;\n margin: 8px 0 24px 0;\n }\n}\n\np + p {\n margin-top: 24px;\n}\n\n.format {\n margin: 0 -12px 24px -12px;\n padding: 12px 12px 8px 12px;\n height: 244px;\n\n box-sizing: border-box;\n background: $lighter;\n background-size: cover;\n background-position: left;\n\n color: #444;\n border-radius: 3px;\n font: normal 16px/24px $nav;\n\n h3 {\n margin: 0;\n padding: 0 0 4px 0;\n font: 600 16px/24px $nav;\n text-transform: uppercase;\n letter-spacing: 1px;\n }\n\n p {\n margin-bottom: 8px;\n }\n}\n\n.format.print, .format.pdf {\n background-position: right;\n text-align: right;\n}\n\n.format-info {\n display: inline-block;\n width: $col * 8;\n text-align: left;\n\n table {\n width: 100%;\n border-collapse: collapse;\n\n td + td {\n padding-left: 5px;\n }\n }\n}\n\n.format.print { background-image: url(\"image/format-print.jpg\"); }\n.format.ebook { background-image: url(\"image/format-ebook.jpg\"); }\n.format.pdf { background-image: url(\"image/format-pdf.jpg\"); }\n.format.web { background-image: url(\"image/format-web.jpg\"); }\n\na.action {\n display: block;\n\n margin: 0 0 4px 0;\n padding: 4px 0;\n text-align: center;\n border-radius: 3px;\n background: $primary;\n\n transition: background-color 0.2s ease,\n color 0.2s ease;\n\n font: 400 17px/24px $nav;\n color: white;\n\n small {\n font-size: 14px;\n padding: 4px;\n color: hsla(0, 0, 100%, 0.7);\n transition: color 0.2s ease;\n }\n}\n\na.action:hover {\n background-color: hsl(200, 85%, 55%);\n\n small {\n color: white;\n }\n}\n\n h3 {\n font: italic 24px/24px $serif;\n margin: 12px 0;\n }\n\nimg.author {\n float: left;\n width: 240px;\n margin: 0 12px 0 -12px;\n padding: 12px;\n\n background: $warm-5;\n border-radius: 3px;\n}\n\ndiv.author {\n vertical-align: top;\n margin: 36px 0 0 240px + $col;\n}\n\nfooter {\n position: relative;\n border-top: solid 1px $light;\n color: $gray-4;\n font: 400 15px $nav;\n text-align: center;\n margin: 12px 0 36px 0;\n padding-top: 48px;\n\n a, a:hover {\n border: none;\n }\n}\n\n@media only screen and (max-width: 700px) {\n main {\n margin: 0 24px;\n }\n\n header {\n margin-bottom: 24px;\n }\n\n img.big {\n display: none;\n }\n\n img.small {\n display: block;\n }\n\n div.intro {\n display: block;\n\n blockquote {\n display: block;\n font: italic 24px/36px $serif;\n }\n\n div.text {\n display: block;\n margin: 24px 0 24px 0;\n }\n }\n\n .format {\n margin-bottom: 12px;\n height: auto;\n background-blend-mode: lighten;\n }\n\n .format-info {\n display: block;\n width: 100%;\n }\n\n .format.print { background-color: #a6a29f; }\n .format.ebook { background-color: #97a2aa; }\n .format.pdf { background-color: #cfccca; }\n .format.web { background-color: #d6dbd3; }\n\n img.author {\n float: none;\n }\n\n div.author {\n margin: 0 0 0 0;\n }\n}\n" }, { "path": "asset/mustache/contents-nav.html", "content": "

  Table of Contents

\n\n{{> prev-next }}\n" }, { "path": "asset/mustache/contents-part.html", "content": "

{{ number }}.{{ title }}

\n" }, { "path": "asset/mustache/contents.html", "content": "{{> header }}\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n

{{title}}

\n\n
\n
\n
\n

❧Frontmatter

\n \n\n {{# part_1 }}\n {{> contents-part }}\n {{/ part_1 }}\n {{# part_2 }}\n {{> contents-part }}\n {{/ part_2 }}\n
\n
\n {{# part_3 }}\n {{> contents-part }}\n {{/ part_3 }}\n\n

❧Backmatter

\n \n
\n
\n
\n\n\n
\n\n{{> footer }}\n" }, { "path": "asset/mustache/footer.html", "content": "
\n\n\n" }, { "path": "asset/mustache/header.html", "content": "\n\n\n\n{{title}} · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "asset/mustache/in_design.html", "content": "\n{{ number }}\n{{ title }}\n{{ part }}\n{{{ body }}}\n\n" }, { "path": "asset/mustache/index.html", "content": "\n\n\n\nCrafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n\n
\n \"Crafting\"Crafting\n
\n\n
\n\n
\n\n

Ever wanted to make your own programming language or wondered how\nthey are designed and built?

If so, this book is for you.

\n\n
\n\n

Crafting Interpreters contains everything you need to implement a\nfull-featured, efficient scripting language. You’ll learn both high-level\nconcepts around parsing and semantics and gritty details like bytecode\nrepresentation and garbage collection. Your brain will light up with new ideas,\nand your hands will get dirty and calloused. It’s a blast.

\n\n

Starting from main(), you build a language that features rich\nsyntax, dynamic typing, garbage collection, lexical scope, first-class\nfunctions, closures, classes, and inheritance. All packed into a few thousand\nlines of clean, fast code that you thoroughly understand because you write each\none yourself.

\n\n

The book is available in four delectable formats:

\n\n
\n\n
\n\n
\n
\n

Print

\n

640 pages of beautiful typography and high resolution hand-drawn\n illustrations. Each page lovingly typeset by the author. The premiere reading\n experience.

\n \n \n \n \n \n \n \n \n \n \n \n \n
\n Amazon.com\n \n .ca\n \n .uk\n \n .au\n \n .de\n \n .fr\n \n .es\n \n .it\n \n .jp\n
\n \n \n \n \n \n
\n Barnes and Noble\n \n Book Depository\n
\n Download Sample PDF\n
\n
\n
\n
\n

eBook

\n

Carefully tuned CSS fits itself to your ebook reader and screen size.\n Full-color syntax highlighting and live hyperlinks. Like Alan Kay's Dynabook\n but real.

\n \n \n \n \n \n \n \n \n \n
\n Kindle Amazon.com\n \n .uk\n \n .ca\n \n .au\n \n .de\n \n .in\n
\n \n \n \n \n \n \n \n \n \n \n
\n .fr\n \n .es\n \n .it\n \n .jp\n \n .br\n \n .mx\n \n Apple Books\n
\n \n \n \n \n \n \n
\n Play Books Google\n \n Nook B&N\n \n EPUB Smashwords\n
\n
\n
\n
\n
\n

PDF

\n

Perfectly mirrors the hand-crafted typesetting and sharp illustrations of\n the print book, but much easier to carry around.

\n Buy from Payhip\n Download Free Sample\n
\n
\n
\n
\n

Web

\n

Meticulous responsive design looks great from your desktop down to your\n phone. Every chapter, aside, and illustration is there. Read the whole book\n for free. Really.

\n Read Now\n
\n
\n\n\n\n
\n

About Robert Nystrom

\n\n

I got bitten by the language bug years ago while on paternity leave between\nmidnight feedings. I cobbled together a number of hobby languages before worming my way into an honest-to-God,\nfull-time programming language job. Today, I work at Google on the Dart language.

\n\n

Before I fell in love with languages, I developed games at Electronic Arts\nfor eight years. I wrote the best-selling book Game Programming\nPatterns based on what I learned there. You can read that book for free\ntoo.

\n\n

If you want more, you can find me on Twitter (@munificentbob), email me at bob\nat this site's domain (though I am slow to respond), read my blog, or join\nmy low frequency mailing list:

\n\n
\n \n
\n
\n \n \n
\n \n
\n
\n \n
\n\n
\n\n\n
\n
\n\n\n" }, { "path": "asset/mustache/nav.html", "content": "{{# is_chapter }}\n

{{ title }}{{ number }}

\n\n\n\n{{/ is_chapter }}\n{{# is_part }}\n

{{ number }}{{ title }}

\n\n\n\n{{/ is_part }}\n{{# is_frontmatter }}\n

{{ number }}{{ title }}

\n
\n{{/ is_frontmatter }}\n\n{{> prev-next }}" }, { "path": "asset/mustache/page.html", "content": "{{> header }}\n\n\n\n\n
\n
\n\n
\n\n
\n\n{{# has_number }}\n
{{ number }}
\n{{/ has_number }}\n{{# is_chapter }}\n

{{ title }}

\n{{/ is_chapter }}\n{{^ is_chapter }}\n

{{ title }}

\n{{/ is_chapter }}\n\n{{{ body }}}\n\n
\n\n{{> footer }}\n" }, { "path": "asset/mustache/prev-next.html", "content": "
\n {{# has_prev }}\n ← Previous\n {{/ has_prev }}\n {{# has_up }}\n ↑ Up\n {{/ has_up }}\n {{# has_next }}\n Next →\n {{/ has_next }}\n
" }, { "path": "asset/sass/chapter.scss", "content": "article.chapter {\n h2 {\n font: 600 30px/24px $serif;\n margin: 69px 0 0 0;\n padding-bottom: 3px;\n\n small {\n font: 800 22px/24px $serif;\n float: right;\n }\n }\n\n h3 {\n font: italic 24px/24px $serif;\n margin: 71px 0 0 0;\n padding-bottom: 1px;\n\n small {\n font: 600 16px/24px $serif;\n float: right;\n }\n }\n\n h2 a, h3 a {\n color: #222;\n border-bottom: none;\n }\n\n h2 a:hover, h3 a:hover {\n border-bottom: none;\n color: inherit;\n }\n\n h2 a::before, h3 a::before {\n position: absolute;\n left: -$col;\n width: $col;\n content: \"\\000A7\";\n color: #fff;\n transition: color 0.2s ease;\n text-align: center;\n }\n\n h2 a:hover::before, h3 a:hover::before {\n color: #ddd;\n }\n\n .challenges, .design-note {\n border-radius: 3px;\n padding: 12px;\n margin: -2px -12px 26px -12px;\n\n font: normal 16px/24px $nav;\n color: #444;\n\n h2 {\n margin: 0 0 -12px 0;\n padding: 0;\n font: 600 16px/24px $nav;\n text-transform: uppercase;\n letter-spacing: 1px;\n }\n\n h2 a {\n color: inherit;\n }\n\n h2 a::before {\n content: none;\n }\n\n ol {\n padding: 0 0 0 18px;\n\n li {\n padding: 0 0 0 6px;\n font-weight: 600;\n\n p {\n font-weight: 400;\n }\n }\n }\n\n pre {\n margin: 0;\n }\n\n // Chapter 23 has some blockquotes in the design note.\n > blockquote {\n p {\n margin: 0 24px;\n font: italic 16px/24px $nav;\n color: #444;\n }\n\n &::before, &::after {\n content: none;\n }\n }\n\n // Use the regular code colors in asides, and not the tinted versions used\n // inside the challenge or design notes boxes themselves.\n aside {\n code, .codehilite {\n color: $warm-dark;\n background: $warm-light;\n }\n }\n\n // Remove the extra padding at the bottom of the box.\n *:last-child {\n margin-bottom: 0;\n }\n }\n\n .challenges .codehilite,\n .design-note .codehilite {\n margin: -12px 0 -12px 0;\n }\n\n .challenges {\n background: $lighter;\n\n code, .codehilite {\n background: hsl(195, 30%, 92%);\n }\n }\n\n .design-note {\n background: hsl(80, 30%, 96%);\n\n code, .codehilite {\n background: hsl(80, 20%, 93%);\n }\n }\n\n table {\n width: 100%;\n border-collapse: collapse;\n\n thead {\n font: 700 15px $serif;\n }\n\n td {\n border-bottom: solid 1px $light;\n line-height: 22px;\n padding: 3px 0 0 0;\n margin: 0;\n }\n\n td + td {\n padding-left: 12px;\n }\n }\n}\n\n// Tablets and mobile go single column.\n@media only screen and (max-width: $col * 20) {\n article.chapter {\n\n // Now that the asides are inline, make them match the challenge/design-note\n // colors and font.\n .challenges, .design-note {\n aside {\n font: normal 15px/24px $nav;\n padding-bottom: 4px;\n }\n }\n\n .challenges {\n aside {\n code, .codehilite {\n background: hsl(195, 30%, 92%);\n }\n }\n }\n\n .design-note {\n aside {\n code, .codehilite {\n background: hsl(80, 20%, 93%);\n }\n }\n }\n }\n}\n\n// Then bring the margins in some.\n// The cut-off sizes here are based on trying to get 72 columns of code to fit.\n@media only screen and (max-width: 630px) {\n article.chapter {\n h2 a::before, h3 a::before {\n left: -($col / 2);\n width: $col / 2;\n }\n }\n}\n\n// Finally start shrinking text.\n@media only screen and (max-width: 580px) {\n article.chapter {\n h2 {\n margin-top: 64px;\n padding-bottom: 2px;\n font-size: 22px;\n line-height: 22px;\n }\n\n h3 {\n margin-top: 64px;\n padding-bottom: 0;\n font-size: 20px;\n }\n\n .challenges, .design-note {\n padding: 11px 11px 8px 11px;\n margin: 25px 0 0 0;\n\n font-size: 15px;\n line-height: 22px;\n\n code, .codehilite {\n font-size: 14px;\n }\n\n h2 {\n padding: 5px 0 4px 6px;\n font-size: 17px;\n line-height: 22px;\n }\n\n aside {\n line-height: 22px;\n }\n }\n }\n}\n" }, { "path": "asset/sass/contents.scss", "content": "article.contents {\n h2 {\n margin: 22px 0 6px 0;\n font: 600 normal 18px/24px $nav;\n text-transform: uppercase;\n letter-spacing: 1px;\n\n .num {\n display: inline-block;\n width: 36px;\n }\n }\n\n ul {\n margin: -12px 0 0 0;\n padding: 6px 0 14px 0;\n }\n\n li {\n padding: 12px 0 0 36px;\n font: normal 16px/24px $nav;\n color: $gray-4;\n\n list-style-type: none;\n\n .num {\n display: inline-block;\n letter-spacing: 1px;\n width: 36px;\n }\n\n a {\n font: 600 17px/24px $nav;\n }\n }\n\n li.design-note {\n padding-top: 0;\n\n a {\n font: 400 16px/23px $nav;\n }\n }\n\n // Format the chapter list in two columns.\n .chapters {\n display: table;\n width: $col * 18;\n }\n\n .row {\n display: table-row;\n }\n\n .first, .second {\n display: table-cell;\n vertical-align: top;\n }\n\n .second {\n padding-left: $col;\n }\n\n footer {\n width: $col * 18;\n }\n}\n\n// Go single-column with the chapter list.\n@media only screen and (max-width: 1344px) {\n article.contents {\n .chapters, .row, .first, .second {\n display: block;\n width: auto;\n }\n\n .second {\n padding-left: 0;\n }\n\n footer {\n width: inherit;\n }\n }\n}\n\n// Then bring the margins in some.\n@media only screen and (max-width: 630px) {\n article.contents {\n h2 .num, li .num {\n width: 28px;\n }\n\n ol, ul {\n margin-left: 0;\n }\n\n li {\n padding-left: 0;\n }\n }\n}\n\n// Finally start shrinking text.\n@media only screen and (max-width: 580px) {\n article.contents {\n h2 {\n margin: 19px 0 6px 0;\n font-size: 17px;\n line-height: 22px;\n }\n\n h3 {\n padding: 1px 0 2px 0;\n font-size: 17px;\n line-height: 22px;\n }\n\n p {\n font-size: 15px;\n line-height: 22px;\n }\n\n ol, ul {\n padding-bottom: 8px;\n }\n\n li {\n font-size: 14px;\n line-height: 22px;\n padding: 4px 0 3px 0;\n }\n }\n}\n" }, { "path": "asset/sass/print.scss", "content": "@import 'shared';\n\n@media print {\n // Pure black text.\n body, a, code {\n color: #000 !important;\n background: none !important;\n }\n\n // Hide non-content stuff.\n nav, .sign-up {\n display: none;\n }\n\n // Get rid of extra margins. The page margin will handle this.\n .page {\n margin: 0 !important;\n }\n\n // Tweak how code is formatted since we don't want to use a background color.\n .codehilite {\n pre {\n color: #000 !important;\n }\n\n margin: 0 !important;\n\n // Borders above and below and no background.\n background: none !important;\n border-radius: 0 !important;\n border-left: solid 1px $warm-4;\n border-right: solid 1px $warm-4;\n\n // Show thicker borders on the left and right instead of a background.\n .insert {\n border-left: solid 3px $warm-4 !important;\n border-right: solid 3px $warm-4 !important;\n background: none !important;\n }\n\n .delete {\n -webkit-print-color-adjust: exact;\n color-adjust: exact;\n }\n\n // Browsers don't honor the specific authored colors when printing if the\n // color is too close the background. Tell the browser not to do that.\n .insert-before span, .insert-after span {\n -webkit-print-color-adjust: exact;\n color-adjust: exact;\n }\n }\n}\n" }, { "path": "asset/sass/shared.scss", "content": "// Font stacks.\n$serif: \"Crimson\", Georgia, serif;\n$mono: \"Source Code Pro\", Menlo, Consolas, Monaco, monospace;\n$nav: \"Source Sans Pro\", sans-serif;\n\n// The main intense primary accent color.\n$primary: hsl(200, 80%, 40%);\n$primary-dark: hsl(200, 100%, 20%);\n$primary-light: hsl(200, 70%, 60%);\n\n// A ramp of washed out blues from dark to light.\n$dark: hsl(215, 20%, 20%);\n$gray-1: hsl(212, 23%, 30%);\n$gray-2: hsl(209, 26%, 40%);\n$gray-3: hsl(206, 30%, 50%);\n$gray-4: hsl(203, 30%, 60%);\n$light: hsl(195, 30%, 90%);\n$lighter: hsl(195, 35%, 95%);\n\n// An opposing warm light color (code background).\n$warm-dark: hsl(40, 0%, 35%);\n$warm-light: hsl(40, 30%, 97%);\n$warm-1: mix($warm-light, $warm-dark, 15%);\n$warm-2: mix($warm-light, $warm-dark, 40%);\n$warm-3: mix($warm-light, $warm-dark, 60%);\n$warm-4: mix($warm-light, $warm-dark, 80%);\n$warm-5: hsl(40, 20%, 95%);\n\n// The full-size design is 28 units wide, in three columns:\n// [][][][][][][][][][][][][][][][][][][][][][][][][][][][]\n// ( 5 ) ( 12 ) ( 6 )\n// They are asymmetric because the left column has a dark background, which\n// requires a double margin.\n$col: 48px;\n\n@font-face {\n font-family: 'Crimson';\n src: url('font/crimson-roman.woff') format('woff');\n}\n\n@font-face {\n font-family: 'Crimson';\n src: url('font/crimson-italic.woff') format('woff');\n font-style: italic;\n}\n\n@font-face {\n font-family: 'Crimson';\n src: url('font/crimson-semibold.woff') format('woff');\n font-weight: 600;\n}\n\n@font-face {\n font-family: 'Crimson';\n src: url('font/crimson-semibolditalic.woff') format('woff');\n font-style: italic;\n font-weight: 600;\n}\n\n@font-face {\n font-family: 'Crimson';\n src: url('font/crimson-bold.woff') format('woff');\n font-weight: bold;\n}\n\n@font-face {\n font-family: 'Crimson';\n src: url('font/crimson-bolditalic.woff') format('woff');\n font-style: italic;\n font-weight: bold;\n}\n\n// Reset stuff.\n\nbody, h1, h2, h3, h4, p, blockquote, code, ul, ol, dl, dd, img {\n margin: 0;\n}\n\nimg {\n outline: none;\n}\n\nimg.arrow {\n width: auto;\n height: 11px;\n}\n\nimg.dot {\n width: auto;\n height: 18px;\n vertical-align: text-bottom;\n}\n\n// Basic styles.\n\nbody {\n color: #222;\n font: normal 16px/24px $serif;\n}\n" }, { "path": "asset/sass/sign-up.scss", "content": ".sign-up {\n padding: 12px;\n margin: 24px 0 24px 0;\n background: hsl(40, 80%, 95%);\n color: hsl(40, 50%, 50%);\n border-radius: 3px;\n\n form {\n display: flex;\n }\n\n input {\n padding: 4px;\n font: 16px $nav;\n outline: none;\n border-radius: 3px;\n border: solid 2px hsl(40, 100%, 75%);\n color: hsl(40, 70%, 30%);\n height: 32px;\n }\n\n input.email {\n display: block;\n box-sizing: border-box;\n width: 100%;\n }\n\n input.button {\n margin-left: 8px;\n padding: 4px 8px;\n font: 600 13px $nav;\n text-transform: uppercase;\n letter-spacing: 1px;\n background: hsl(40, 100%, 60%);\n border: none;\n\n transition: background-color 0.2s ease;\n }\n\n input.button:hover {\n background: hsl(40, 100%, 75%);\n }\n\n input:focus {\n border-color: hsl(40, 100%, 50%);\n }\n}" }, { "path": "asset/style.scss", "content": "@import 'sass/shared';\n@import 'sass/chapter';\n@import 'sass/contents';\n@import 'sass/sign-up';\n@import 'sass/print';\n\n// Make sure we don't split on the thin spaces around an em dash.\n.emdash {\n white-space: nowrap;\n}\n\n.scrim {\n position: absolute;\n width: 100%;\n height: 10000px;\n\n z-index: 4;\n\n // background: url('columns.png');\n background: url('rows.png');\n}\n\n// Used for drawing the bitwise operators \"AND\", \"OR\", and \"NOT\" in small caps.\n.small-caps {\n font-weight: 600;\n font-size: 13px;\n}\n\na {\n color: $primary;\n text-decoration: none;\n\n border-bottom: solid 1px transparentize($light, 1.0);\n\n transition: color 0.2s ease,\n border-color 0.4s ease;\n}\n\na:hover {\n color: $primary;\n border-bottom: solid 1px opacify($light, 1.0);\n}\n\nnav {\n font: 300 15px/24px $nav;\n background: $dark;\n color: $gray-2;\n\n a, h2 a {\n color: $gray-4;\n text-decoration: none;\n border-bottom: none;\n }\n\n a:hover {\n color: $light;\n text-decoration: none;\n border-bottom: none;\n }\n\n img {\n box-sizing: border-box;\n width: 100%;\n padding: 55px $col 23px $col;\n }\n\n h2 {\n font: 400 16px/24px $nav;\n text-transform: uppercase;\n letter-spacing: 1px;\n color: $gray-4;\n }\n\n h3 {\n font: 400 18px/24px $nav;\n color: $gray-4;\n }\n\n h2 small, h3 small {\n float: right;\n font-size: 16px;\n color: $gray-2;\n }\n\n ol, ul {\n margin: 6px 0 3px 0;\n padding: 6px 0 4px 24px;\n border-top: solid 1px $gray-1;\n border-bottom: solid 1px $gray-1;\n }\n\n ul {\n list-style-type: none;\n padding-left: 0;\n }\n\n hr {\n border: none;\n border-top: solid 1px $gray-1;\n margin: 6px 0 0 0;\n padding: 0 0 3px 0;\n }\n\n li small {\n float: right;\n font-size: 14px;\n color: $gray-2;\n }\n\n li.divider {\n margin: 5px 0 7px 0;\n border-top: solid 1px $gray-1;\n }\n\n li.end-part {\n font-size: 12px;\n font-weight: 400;\n text-transform: uppercase;\n letter-spacing: 1px;\n\n small {\n font-weight: 300;\n text-transform: none;\n letter-spacing: 0;\n }\n }\n\n .prev-next {\n padding-top: 7px;\n font: 400 12px/18px $nav;\n text-align: center;\n text-transform: uppercase;\n letter-spacing: 1px;\n }\n}\n\nnav.wide {\n position: fixed;\n width: $col * 7;\n height: 100%;\n\n .contents {\n margin: 24px $col;\n }\n}\n\n// This is needed to make the nav fixed (not scrolling with the content) but\n// still positioned horizontally based on the page.\n// See: http://stackoverflow.com/a/11833892/9457\n.nav-wrapper {\n position: absolute;\n right: $col * 6;\n}\n\n// For medium-sized screens, the navigation floats over the same column as the\n// asides.\nnav.floating {\n // Only shown on narrower screens.\n display: none;\n\n z-index: 2;\n position: absolute;\n width: $col * 6;\n\n border-bottom-left-radius: 3px;\n border-bottom-right-radius: 3px;\n\n #expand-nav {\n padding: 0 0 4px 0;\n display: block;\n font-size: 20px;\n text-align: center;\n color: $gray-2;\n cursor: pointer;\n\n transition: padding 0.2s ease,\n margin 0.2s ease,\n color 0.2s ease;\n }\n\n #expand-nav, #expand-nav:hover {\n border-bottom: none;\n }\n\n #expand-nav:hover {\n color: $light;\n }\n\n .expandable {\n overflow: hidden;\n padding: 0 12px;\n\n // Using max-height instead of height to allow the list to navigation to\n // automatically choose its height based on the size of the list while\n // still transitioning.\n // See: http://stackoverflow.com/a/8331169/9457\n max-height: 0;\n transition: margin 0.2s ease,\n max-height 1.0s ease;\n\n .prev-next {\n padding-bottom: 6px;\n }\n }\n\n .expandable.shown {\n // This should be as small as possible while still being large enough for\n // the worst case chapter.\n max-height: 550px;\n }\n\n img {\n padding: 110px $col/2 23px $col/2;\n }\n}\n\nnav.floating.pinned {\n position: fixed;\n\n top: -85px;\n\n .expandable {\n margin-top: -13px;\n }\n\n #expand-nav {\n margin-top: -14px;\n }\n}\n\nnav.narrow {\n display: none;\n\n text-align: center;\n\n img {\n box-sizing: content-box;\n padding: 11px 0 3px 0;\n width: auto;\n height: 27px;\n }\n\n .prev, .next {\n font-size: 32px;\n position: absolute;\n top: 12px;\n padding: 0 $col;\n }\n\n .prev {\n left: 0;\n }\n\n .next {\n right: 0;\n }\n}\n\n.left {\n float: left;\n}\n\n.right {\n float: right;\n}\n\n.page {\n position: relative;\n\n width: $col * 19;\n margin: 0 auto 0 $col * 8;\n}\n\n// Make em dashes look pretty. Goals:\n//\n// - Add a tiny bit of space on either side. Completely unspaced em dashes\n// look too tight to me.\n// - Allow an em dash at the end of a line.\n// - Prevent an em dash at the beginning of a line.\n//\n// Wrapping each `—` in a span with this class and consuming the\n// preceding whitespace seems to accomplish that.\n.em {\n padding: 0 .1em;\n white-space: nowrap;\n}\n\n// Make ellipses follow Chicago style. The `…` entity puts a tiny amount\n// of space between each `.`, but not as much as Chicago style specificies. It\n// also doesn't put any space before. Instead, the build system writes a span\n// of this class with thin-space separated dots. This class here ensures there\n// is no splitting between the dots.\n.ellipse {\n white-space: nowrap;\n}\n\ncode {\n font: normal 16px $mono;\n color: $warm-1;\n white-space: nowrap;\n padding: 2px;\n}\n\nstrong code {\n font-weight: bold;\n color: inherit;\n}\n\na code {\n color: $primary;\n}\n\n.codehilite {\n color: $warm-dark;\n background: $warm-light;\n border-radius: 3px;\n padding: 12px;\n margin: -12px;\n}\n\npre {\n font: normal 13px/20px $mono;\n margin: 0;\n padding: 0;\n\n // If the code doesn't fit, just force it to wrap instead of cropping it. It\n // doesn't look great, but it ensures the code is visible and can be correctly\n // copy-pasted.\n white-space: pre-wrap;\n overflow-wrap: anywhere;\n}\n\n// If the chapter ends with code, don't overlap the challenges box.\ndiv.codehilite + div.challenges {\n margin-top: $col / 2;\n}\n\narticle {\n position: relative;\n width: $col * 12;\n\n h1 {\n position: relative;\n font: 48px/48px $serif;\n padding: 109px 0 19px 0;\n z-index: 2;\n }\n\n h1.part {\n font: 600 36px/48px $nav;\n padding: 108px 0 20px 0;\n text-transform: uppercase;\n letter-spacing: 1px;\n }\n\n .number {\n position: absolute;\n top: 50px;\n left: $col * 13;\n\n z-index: 1;\n\n font: 300 96px $nav;\n color: $light;\n }\n\n p {\n margin: 24px 0;\n }\n\n ol, ul {\n margin: 24px 0;\n padding: 0 0 0 24px;\n }\n\n img {\n max-width: 100%;\n }\n\n img.wide {\n max-width: none;\n width: $col * 19;\n }\n}\n\naside {\n position: absolute;\n right: -$col * 7;\n width: $col * 6;\n\n font: normal 14px/20px $serif;\n\n border-top: solid 1px $light;\n\n p {\n margin: 20px 0;\n }\n\n p:first-child,\n img:first-child {\n margin-top: 4px;\n }\n\n p:last-child {\n margin-bottom: 4px;\n }\n\n code {\n font-size: 14px;\n border-radius: 2px;\n padding: 1px 2px;\n }\n\n .codehilite {\n padding: 6px;\n margin: -12px 0;\n }\n\n .codehilite:last-child {\n margin-bottom: 4px;\n }\n\n img.above {\n position: absolute;\n bottom: 100%;\n margin-bottom: 16px;\n }\n\n blockquote {\n margin: 20px 0;\n\n &::before, &::after {\n content: none;\n }\n\n p {\n margin: 0 12px;\n font: italic 15px/20px $serif;\n color: inherit;\n }\n }\n}\n\n// Sometimes there isn't room to hang the aside *down* next to the content it's\n// annotating, so support asides where the bottom is aligned with the content.\naside.bottom {\n border-top: none;\n border-bottom: solid 1px $light;\n}\n\nblockquote {\n position: relative;\n\n margin: 29px 0 31px 0;\n\n &::before, &::after {\n position: absolute;\n top: -20px;\n font: italic 72px $serif;\n color: $light;\n }\n\n &::before {\n content: \"\\201c\";\n left: -7px;\n }\n\n &::after {\n content: \"\\201d\";\n right: 8px;\n }\n\n p {\n margin: 0 $col;\n font: italic 24px/36px $serif;\n color: $gray-3;\n \n em {\n font-style: normal;\n }\n }\n\n cite {\n display: block;\n text-align: right;\n color: $gray-4;\n font-style: normal;\n font-size: 18px;\n\n &::before {\n content: \"\\2014\\00a0\";\n color: $light;\n }\n\n em {\n font-style: italic;\n }\n }\n}\n\nfooter {\n position: relative;\n border-top: solid 1px $light;\n color: $gray-4;\n font: 400 15px $nav;\n text-align: center;\n margin: 48px 0;\n padding-top: 48px;\n\n a, a:hover {\n border: none;\n }\n\n .next {\n position: absolute;\n right: 0;\n top: -13px;\n\n padding-left: 4px;\n background: #fff;\n\n font: 400 17px/24px $nav;\n text-transform: uppercase;\n letter-spacing: 1px;\n }\n\n .next:hover {\n color: $primary-dark;\n border: none;\n }\n}\n\n.dedication {\n margin: 96px 0 128px 0;\n text-align: center;\n\n img {\n width: 50%;\n }\n}\n\n.source-file, .source-file-narrow {\n font: normal 11px/16px $mono;\n color: $warm-3;\n\n em {\n color: $warm-2;\n font-style: normal;\n }\n}\n\n.source-file-narrow {\n // Don't show unless in single-column.\n display: none;\n\n margin: 0px -12px 0 0;\n padding: 14px 0 0 0;\n text-align: right;\n}\n\n.source-file {\n position: absolute;\n right: -$col * 7;\n width: $col * 6;\n padding: 2px 0 0 0;\n\n &::before {\n content: \"<<\";\n color: $warm-4;\n position: absolute;\n left: -($col - 12px);\n width: $col - 12px;\n text-align: center;\n }\n}\n\n// Syntax highlighting.\n.codehilite {\n pre { color: mix($warm-light, $warm-dark, 20%); }\n\n .k { color: hsl(200, 100%, 45%); } // Keyword.\n .n { color: hsl( 20, 70%, 55%); } // Number.\n .s { color: hsl( 40, 70%, 45%); } // String.\n .e { color: hsl( 45, 80%, 55%); } // String escape.\n .c { color: mix($warm-light, $warm-dark, 50%); } // Comment.\n .a { color: hsl(270, 50%, 60%); } // Preprocessor, annotation.\n .i { color: hsl(200, 70%, 35%); } // Identifier.\n .t { color: hsl(185, 100%, 35%); } // Type name.\n\n .insert {\n margin: -2px -12px;\n padding: 2px 10px;\n border-left: solid 2px $warm-4;\n border-right: solid 2px $warm-4;\n background: $warm-5;\n }\n\n .delete {\n margin: -2px -12px;\n padding: 2px 10px;\n border-left: solid 2px $warm-4;\n border-right: solid 2px $warm-4;\n // Hatched lines.\n background: repeating-linear-gradient(\n -45deg,\n $warm-4,\n $warm-4 1px,\n rgba(0, 0, 0, 0.0) 1px,\n rgba(0, 0, 0, 0.0) 6px\n );\n\n span {\n color: $warm-3;\n }\n }\n\n // Snippets of code before and after real code to show where to insert it.\n .insert-before, .insert-after {\n color: $warm-3;\n }\n\n // When we just add a trailing comma to a line, highlight it specially.\n .insert-before .insert-comma {\n margin: -2px -1px;\n padding: 2px 1px;\n border-radius: 2px;\n\n background: $warm-5;\n color: $warm-dark;\n }\n}\n\n// On a not-entirely-large screen, don't show the fixed nav on the left.\n@media only screen and (max-width: 1344px) {\n nav.wide { display: none; }\n nav.floating { display: block; }\n\n body {\n margin: 0 24px;\n }\n\n .page {\n position: relative;\n width: inherit;\n max-width: $col * 19;\n margin: 0 auto;\n }\n\n article {\n width: inherit;\n margin-right: $col * 7;\n\n // Move the number over to not be hidden behind the navigation.\n .number {\n top: 73px;\n left: inherit;\n right: 0;\n font-size: 72px;\n }\n\n h1 {\n padding: 110px 0 18px 0;\n font-size: 44px;\n }\n }\n}\n\n// Tablets and mobile go single column.\n@media only screen and (max-width: $col * 20) {\n body {\n margin: 0;\n }\n\n nav.floating {\n display: none;\n }\n\n nav.narrow {\n display: block;\n }\n\n .page {\n margin: 0 $col;\n width: inherit;\n }\n\n article {\n margin: 0;\n\n // Size wide images to fit inside the column again.\n img.wide {\n width: inherit;\n max-width: 100%;\n }\n }\n\n aside {\n position: inherit;\n right: inherit;\n width: inherit;\n\n border-bottom: solid 1px $light;\n\n p:first-child {\n margin-top: 8px;\n }\n\n p:last-child {\n margin-bottom: 8px;\n }\n\n // If an aside ends with code (like in \"classes.html\"), then make sure we\n // give it some margin.\n div.codehilite:last-child {\n margin-bottom: 12px;\n }\n\n // Make sure aside images don't get too big when the asides are inlined\n // in single column mode.\n img {\n display: block;\n max-width: $col * 6;\n margin: 0 auto;\n }\n\n img.above {\n position: relative;\n }\n }\n\n // If aside is right before a code block (when the asides are inline), make\n // sure they don't overlap.\n aside + div.codehilite {\n margin-top: 12px;\n }\n\n div.codehilite + aside {\n margin-top: 24px;\n }\n\n .source-file {\n display: none;\n }\n\n .source-file-narrow {\n display: block;\n }\n}\n\n// Then bring the margins in some.\n// The cut-off sizes here are based on trying to get 72 columns of code to fit.\n@media only screen and (max-width: 630px) {\n .page {\n margin: 0 $col / 2;\n width: inherit;\n }\n\n nav.narrow {\n .prev, .next {\n padding: 0 $col / 2;\n }\n }\n}\n\n// Finally, shrink the grid to 22px and shrink the text.\n@media only screen and (max-width: 580px) {\n body {\n font-size: 15px;\n line-height: 22px;\n }\n\n .small-caps {\n font-size: 12px;\n }\n\n .scrim {\n background: url('rows-22.png');\n }\n\n nav.narrow {\n img {\n padding: 9px 0 1px 0;\n height: 27px;\n }\n\n .prev, .next {\n top: 11px;\n }\n }\n\n article {\n h1 {\n font-size: 36px;\n padding: 100px 0 14px 0;\n }\n\n h1.part {\n font-size: 30px;\n padding: 97px 0 17px 0;\n }\n\n .number {\n top: 61px;\n font-size: 72px;\n }\n\n p {\n margin: 22px 0;\n }\n\n ol, ul {\n margin: 22px 0;\n padding: 0 0 0 22px;\n }\n }\n\n blockquote {\n margin: 27px 0 28px 0;\n\n &::before, &::after {\n top: -17px;\n font-size: 52px;\n }\n\n p {\n margin: 0 22px;\n font-size: 20px;\n line-height: 33px;\n }\n }\n\n footer {\n .next {\n font-size: 15px;\n }\n }\n}\n" }, { "path": "book/a-bytecode-virtual-machine.md", "content": "Our Java interpreter, jlox, taught us many of the fundamentals of programming\nlanguages, but we still have much to learn. First, if you run any interesting\nLox programs in jlox, you'll discover it's achingly slow. The style of\ninterpretation it uses -- walking the AST directly -- is good enough for *some*\nreal-world uses, but leaves a lot to be desired for a general-purpose scripting\nlanguage.\n\nAlso, we implicitly rely on runtime features of the JVM itself. We take for\ngranted that things like `instanceof` in Java work *somehow*. And we never for a\nsecond worry about memory management because the JVM's garbage collector takes\ncare of it for us.\n\nWhen we were focused on high-level concepts, it was fine to gloss over those.\nBut now that we know our way around an interpreter, it's time to dig down to\nthose lower layers and build our own virtual machine from scratch using nothing\nmore than the C standard library...\n" }, { "path": "book/a-map-of-the-territory.md", "content": "> You must have a map, no matter how rough. Otherwise you wander all over the\n> place. In *The Lord of the Rings* I never made anyone go farther than he could\n> on a given day.\n>\n> J. R. R. Tolkien\n\nWe don't want to wander all over the place, so before we set off, let's scan\nthe territory charted by previous language implementers. It will help us\nunderstand where we are going and the alternate routes others have taken.\n\nFirst, let me establish a shorthand. Much of this book is about a language's\n*implementation*, which is distinct from the *language itself* in some sort of\nPlatonic ideal form. Things like \"stack\", \"bytecode\", and \"recursive descent\",\nare nuts and bolts one particular implementation might use. From the user's\nperspective, as long as the resulting contraption faithfully follows the\nlanguage's specification, it's all implementation detail.\n\nWe're going to spend a lot of time on those details, so if I have to write\n\"language *implementation*\" every single time I mention them, I'll wear my\nfingers off. Instead, I'll use \"language\" to refer to either a language or an\nimplementation of it, or both, unless the distinction matters.\n\n## The Parts of a Language\n\nEngineers have been building programming languages since the Dark Ages of\ncomputing. As soon as we could talk to computers, we discovered doing so was too\nhard, and we enlisted their help. I find it fascinating that even though today's\nmachines are literally a million times faster and have orders of magnitude more\nstorage, the way we build programming languages is virtually unchanged.\n\nThough the area explored by language designers is vast, the trails they've\ncarved through it are few. Not every language takes the\nexact same path -- some take a shortcut or two -- but otherwise they are\nreassuringly similar, from Rear Admiral Grace Hopper's first COBOL compiler all\nthe way to some hot, new, transpile-to-JavaScript language whose \"documentation\"\nconsists entirely of a single, poorly edited README in a Git repository\nsomewhere.\n\n\n\nI visualize the network of paths an implementation may choose as climbing a\nmountain. You start off at the bottom with the program as raw source text,\nliterally just a string of characters. Each phase analyzes the program and\ntransforms it to some higher-level representation where the semantics -- what\nthe author wants the computer to do -- become more apparent.\n\nEventually we reach the peak. We have a bird's-eye view of the user's program\nand can see what their code *means*. We begin our descent down the other side of\nthe mountain. We transform this highest-level representation down to\nsuccessively lower-level forms to get closer and closer to something we know how\nto make the CPU actually execute.\n\n\"The\n\nLet's trace through each of those trails and points of interest. Our journey\nbegins on the left with the bare text of the user's source code:\n\n\"var\n\n### Scanning\n\nThe first step is **scanning**, also known as **lexing**, or (if you're trying\nto impress someone) **lexical analysis**. They all mean pretty much the same\nthing. I like \"lexing\" because it sounds like something an evil supervillain\nwould do, but I'll use \"scanning\" because it seems to be marginally more\ncommonplace.\n\nA **scanner** (or **lexer**) takes in the linear stream of characters and chunks\nthem together into a series of something more akin to \"words\". In programming languages, each of these words is\ncalled a **token**. Some tokens are single characters, like `(` and `,`. Others\nmay be several characters long, like numbers (`123`), string literals (`\"hi!\"`),\nand identifiers (`min`).\n\n\n\nSome characters in a source file don't actually mean anything. Whitespace is\noften insignificant, and comments, by definition, are ignored by the language.\nThe scanner usually discards these, leaving a clean sequence of meaningful\ntokens.\n\n\"[var]\n\n### Parsing\n\nThe next step is **parsing**. This is where our syntax gets a **grammar** -- the\nability to compose larger expressions and statements out of smaller parts. Did\nyou ever diagram sentences in English class? If so, you've done what a parser\ndoes, except that English has thousands and thousands of \"keywords\" and an\noverflowing cornucopia of ambiguity. Programming languages are much simpler.\n\nA **parser** takes the flat sequence of tokens and builds a tree structure that\nmirrors the nested nature of the grammar. These trees have a couple of different\nnames -- **parse tree** or **abstract syntax tree** -- depending on how\nclose to the bare syntactic structure of the source language they are. In\npractice, language hackers usually call them **syntax trees**, **ASTs**, or\noften just **trees**.\n\n\"An\n\nParsing has a long, rich history in computer science that is closely tied to the\nartificial intelligence community. Many of the techniques used today to parse\nprogramming languages were originally conceived to parse *human* languages by AI\nresearchers who were trying to get computers to talk to us.\n\nIt turns out human languages were too messy for the rigid grammars those parsers\ncould handle, but they were a perfect fit for the simpler artificial grammars of\nprogramming languages. Alas, we flawed humans still manage to use those simple\ngrammars incorrectly, so the parser's job also includes letting us know when we\ndo by reporting **syntax errors**.\n\n### Static analysis\n\nThe first two stages are pretty similar across all implementations. Now, the\nindividual characteristics of each language start coming into play. At this\npoint, we know the syntactic structure of the code -- things like which\nexpressions are nested in which -- but we don't know much more than that.\n\nIn an expression like `a + b`, we know we are adding `a` and `b`, but we don't\nknow what those names refer to. Are they local variables? Global? Where are they\ndefined?\n\nThe first bit of analysis that most languages do is called **binding** or\n**resolution**. For each **identifier**, we find out where that name is defined\nand wire the two together. This is where **scope** comes into play -- the region\nof source code where a certain name can be used to refer to a certain\ndeclaration.\n\nIf the language is statically typed, this is when we\ntype check. Once we know where `a` and `b` are declared, we can also figure out\ntheir types. Then if those types don't support being added to each other, we\nreport a **type error**.\n\n\n\nTake a deep breath. We have attained the summit of the mountain and a sweeping\nview of the user's program. All this semantic insight that is visible to us from\nanalysis needs to be stored somewhere. There are a few places we can squirrel it\naway:\n\n* Often, it gets stored right back as **attributes** on the syntax tree\n itself -- extra fields in the nodes that aren't initialized during parsing\n but get filled in later.\n\n* Other times, we may store data in a lookup table off to the side. Typically,\n the keys to this table are identifiers -- names of variables and declarations.\n In that case, we call it a **symbol table** and the values it associates with\n each key tell us what that identifier refers to.\n\n* The most powerful bookkeeping tool is to transform the tree into an entirely\n new data structure that more directly expresses the semantics of the code.\n That's the next section.\n\nEverything up to this point is considered the **front end** of the\nimplementation. You might guess everything after this is the **back end**, but\nno. Back in the days of yore when \"front end\" and \"back end\" were coined,\ncompilers were much simpler. Later researchers invented new phases to stuff\nbetween the two halves. Rather than discard the old terms, William Wulf and\ncompany lumped those new phases into the charming but spatially paradoxical name\n**middle end**.\n\n### Intermediate representations\n\nYou can think of the compiler as a pipeline where each stage's job is to\norganize the data representing the user's code in a way that makes the next\nstage simpler to implement. The front end of the pipeline is specific to the\nsource language the program is written in. The back end is concerned with the\nfinal architecture where the program will run.\n\nIn the middle, the code may be stored in some **intermediate\nrepresentation** (**IR**) that isn't tightly tied to either the source or\ndestination forms (hence \"intermediate\"). Instead, the IR acts as an interface\nbetween these two languages.\n\n\n\nThis lets you support multiple source languages and target platforms with less\neffort. Say you want to implement Pascal, C, and Fortran compilers, and you want\nto target x86, ARM, and, I dunno, SPARC. Normally, that means you're signing up\nto write *nine* full compilers: Pascal→x86, C→ARM, and every other\ncombination.\n\nA shared intermediate representation reduces that\ndramatically. You write *one* front end for each source language that produces\nthe IR. Then *one* back end for each target architecture. Now you can mix and\nmatch those to get every combination.\n\n\n\nThere's another big reason we might want to transform the code into a form that\nmakes the semantics more apparent...\n\n### Optimization\n\nOnce we understand what the user's program means, we are free to swap it out\nwith a different program that has the *same semantics* but implements them more\nefficiently -- we can **optimize** it.\n\nA simple example is **constant folding**: if some expression always evaluates to\nthe exact same value, we can do the evaluation at compile time and replace the\ncode for the expression with its result. If the user typed in this:\n\n```java\npennyArea = 3.14159 * (0.75 / 2) * (0.75 / 2);\n```\n\nwe could do all of that arithmetic in the compiler and change the code to:\n\n```java\npennyArea = 0.4417860938;\n```\n\nOptimization is a huge part of the programming language business. Many language\nhackers spend their entire careers here, squeezing every drop of performance\nthey can out of their compilers to get their benchmarks a fraction of a percent\nfaster. It can become a sort of obsession.\n\nWe're mostly going to hop over that rathole in this\nbook. Many successful languages have surprisingly few compile-time\noptimizations. For example, Lua and CPython generate relatively unoptimized\ncode, and focus most of their performance effort on the runtime.\n\n\n\n### Code generation\n\nWe have applied all of the optimizations we can think of to the user's program.\nThe last step is converting it to a form the machine can actually run. In other\nwords, **generating code** (or **code gen**), where \"code\" here usually refers to\nthe kind of primitive assembly-like instructions a CPU runs and not the kind of\n\"source code\" a human might want to read.\n\nFinally, we are in the **back end**, descending the other side of the mountain.\nFrom here on out, our representation of the code becomes more and more\nprimitive, like evolution run in reverse, as we get closer to something our\nsimple-minded machine can understand.\n\nWe have a decision to make. Do we generate instructions for a real CPU or a\nvirtual one? If we generate real machine code, we get an executable that the OS\ncan load directly onto the chip. Native code is lightning fast, but generating\nit is a lot of work. Today's architectures have piles of instructions, complex\npipelines, and enough historical baggage to fill a 747's\nluggage bay.\n\nSpeaking the chip's language also means your compiler is tied to a specific\narchitecture. If your compiler targets [x86][] machine code, it's not going to\nrun on an [ARM][] device. All the way back in the '60s, during the\nCambrian explosion of computer architectures, that lack of portability was a\nreal obstacle.\n\n\n\n[x86]: https://en.wikipedia.org/wiki/X86\n[arm]: https://en.wikipedia.org/wiki/ARM_architecture\n\nTo get around that, hackers like Martin Richards and Niklaus Wirth, of BCPL and\nPascal fame, respectively, made their compilers produce *virtual* machine code.\nInstead of instructions for some real chip, they produced code for a\nhypothetical, idealized machine. Wirth called this **p-code** for *portable*,\nbut today, we generally call it **bytecode** because each instruction is often a\nsingle byte long.\n\nThese synthetic instructions are designed to map a little more closely to the\nlanguage's semantics, and not be so tied to the peculiarities of any one\ncomputer architecture and its accumulated historical cruft. You can think of it\nlike a dense, binary encoding of the language's low-level operations.\n\n### Virtual machine\n\nIf your compiler produces bytecode, your work isn't over once that's done. Since\nthere is no chip that speaks that bytecode, it's your job to translate. Again,\nyou have two options. You can write a little mini-compiler for each target\narchitecture that converts the bytecode to native code for that machine. You\nstill have to do work for each chip you support, but\nthis last stage is pretty simple and you get to reuse the rest of the compiler\npipeline across all of the machines you support. You're basically using your\nbytecode as an intermediate representation.\n\n\n\nOr you can write a **virtual machine** (**VM**), a\nprogram that emulates a hypothetical chip supporting your virtual architecture\nat runtime. Running bytecode in a VM is slower than translating it to native\ncode ahead of time because every instruction must be simulated at runtime each\ntime it executes. In return, you get simplicity and portability. Implement your\nVM in, say, C, and you can run your language on any platform that has a C\ncompiler. This is how the second interpreter we build in this book works.\n\n\n\n### Runtime\n\nWe have finally hammered the user's program into a form that we can execute. The\nlast step is running it. If we compiled it to machine code, we simply tell the\noperating system to load the executable and off it goes. If we compiled it to\nbytecode, we need to start up the VM and load the program into that.\n\nIn both cases, for all but the basest of low-level languages, we usually need\nsome services that our language provides while the program is running. For\nexample, if the language automatically manages memory, we need a garbage\ncollector going in order to reclaim unused bits. If our language supports\n\"instance of\" tests so you can see what kind of object you have, then we need\nsome representation to keep track of the type of each object during execution.\n\nAll of this stuff is going at runtime, so it's called, appropriately, the\n**runtime**. In a fully compiled language, the code implementing the runtime\ngets inserted directly into the resulting executable. In, say, [Go][], each\ncompiled application has its own copy of Go's runtime directly embedded in it.\nIf the language is run inside an interpreter or VM, then the runtime lives\nthere. This is how most implementations of languages like Java, Python, and\nJavaScript work.\n\n[go]: https://golang.org/\n\n## Shortcuts and Alternate Routes\n\nThat's the long path covering every possible phase you might implement. Many\nlanguages do walk the entire route, but there are a few shortcuts and alternate\npaths.\n\n### Single-pass compilers\n\nSome simple compilers interleave parsing, analysis, and code generation so that\nthey produce output code directly in the parser, without ever allocating any\nsyntax trees or other IRs. These **single-pass\ncompilers** restrict the design of the language. You have no intermediate\ndata structures to store global information about the program, and you don't\nrevisit any previously parsed part of the code. That means as soon as you see\nsome expression, you need to know enough to correctly compile it.\n\n\n\nPascal and C were designed around this limitation. At the time, memory was so\nprecious that a compiler might not even be able to hold an entire *source file*\nin memory, much less the whole program. This is why Pascal's grammar requires\ntype declarations to appear first in a block. It's why in C you can't call a\nfunction above the code that defines it unless you have an explicit forward\ndeclaration that tells the compiler what it needs to know to generate code for a\ncall to the later function.\n\n### Tree-walk interpreters\n\nSome programming languages begin executing code right after parsing it to an AST\n(with maybe a bit of static analysis applied). To run the program, the\ninterpreter traverses the syntax tree one branch and leaf at a time, evaluating\neach node as it goes.\n\nThis implementation style is common for student projects and little languages,\nbut is not widely used for general-purpose languages\nsince it tends to be slow. Some people use \"interpreter\" to mean only these\nkinds of implementations, but others define that word more generally, so I'll\nuse the inarguably explicit **tree-walk interpreter** to refer to these. Our\nfirst interpreter rolls this way.\n\n\n\n### Transpilers\n\nWriting a complete back end for a language can be a lot\nof work. If you have some existing generic IR to target, you could bolt your\nfront end onto that. Otherwise, it seems like you're stuck. But what if you\ntreated some other *source language* as if it were an intermediate\nrepresentation?\n\nYou write a front end for your language. Then, in the back end, instead of doing\nall the work to *lower* the semantics to some primitive target language, you\nproduce a string of valid source code for some other language that's about as\nhigh level as yours. Then, you use the existing compilation tools for *that*\nlanguage as your escape route off the mountain and down to something you can\nexecute.\n\nThey used to call this a **source-to-source compiler** or a **transcompiler**.\nAfter the rise of languages that compile to JavaScript in order to run in the\nbrowser, they've affected the hipster sobriquet **transpiler**.\n\n\n\nWhile the first transcompiler translated one assembly language to another,\ntoday, most transpilers work on higher-level languages. After the viral spread\nof UNIX to machines various and sundry, there began a long tradition of\ncompilers that produced C as their output language. C compilers were available\neverywhere UNIX was and produced efficient code, so targeting C was a good way\nto get your language running on a lot of architectures.\n\nWeb browsers are the \"machines\" of today, and their \"machine code\" is\nJavaScript, so these days it seems [almost every language out there][js] has a\ncompiler that targets JS since that's the main way to get\nyour code running in a browser.\n\n[js]: https://github.com/jashkenas/coffeescript/wiki/list-of-languages-that-compile-to-js\n\n\n\nThe front end -- scanner and parser -- of a transpiler looks like other\ncompilers. Then, if the source language is only a simple syntactic skin over the\ntarget language, it may skip analysis entirely and go straight to outputting the\nanalogous syntax in the destination language.\n\nIf the two languages are more semantically different, you'll see more of the\ntypical phases of a full compiler including analysis and possibly even\noptimization. Then, when it comes to code generation, instead of outputting some\nbinary language like machine code, you produce a string of grammatically correct\nsource (well, destination) code in the target language.\n\nEither way, you then run that resulting code through the output language's\nexisting compilation pipeline, and you're good to go.\n\n### Just-in-time compilation\n\nThis last one is less a shortcut and more a dangerous alpine scramble best\nreserved for experts. The fastest way to execute code is by compiling it to\nmachine code, but you might not know what architecture your end user's machine\nsupports. What to do?\n\nYou can do the same thing that the HotSpot Java Virtual Machine (JVM),\nMicrosoft's Common Language Runtime (CLR), and most JavaScript interpreters do.\nOn the end user's machine, when the program is loaded -- either from source in\nthe case of JS, or platform-independent bytecode for the JVM and CLR -- you\ncompile it to native code for the architecture their computer supports.\nNaturally enough, this is called **just-in-time compilation**. Most hackers just\nsay \"JIT\", pronounced like it rhymes with \"fit\".\n\nThe most sophisticated JITs insert profiling hooks into the generated code to\nsee which regions are most performance critical and what kind of data is flowing\nthrough them. Then, over time, they will automatically recompile those hot spots with more advanced optimizations.\n\n\n\n## Compilers and Interpreters\n\nNow that I've stuffed your head with a dictionary's worth of programming\nlanguage jargon, we can finally address a question that's plagued coders since\ntime immemorial: What's the difference between a compiler and an interpreter?\n\nIt turns out this is like asking the difference between a fruit and a vegetable.\nThat seems like a binary either-or choice, but actually \"fruit\" is a *botanical*\nterm and \"vegetable\" is *culinary*. One does not strictly imply the negation of\nthe other. There are fruits that aren't vegetables (apples) and vegetables that\naren't fruits (carrots), but also edible plants that are both fruits *and*\nvegetables, like tomatoes.\n\n\n\n\"A\n\n\n\nSo, back to languages:\n\n* **Compiling** is an *implementation technique* that involves translating a\n source language to some other -- usually lower-level -- form. When you\n generate bytecode or machine code, you are compiling. When you transpile to\n another high-level language, you are compiling too.\n\n* When we say a language implementation \"is a **compiler**\", we mean it\n translates source code to some other form but doesn't execute it. The user has\n to take the resulting output and run it themselves.\n\n* Conversely, when we say an implementation \"is an **interpreter**\", we mean it\n takes in source code and executes it immediately. It runs programs \"from\n source\".\n\nLike apples and oranges, some implementations are clearly compilers and *not*\ninterpreters. GCC and Clang take your C code and compile it to machine code. An\nend user runs that executable directly and may never even know which tool was\nused to compile it. So those are *compilers* for C.\n\nIn older versions of Matz's canonical implementation of Ruby, the user ran Ruby\nfrom source. The implementation parsed it and executed it directly by traversing\nthe syntax tree. No other translation occurred, either internally or in any\nuser-visible form. So this was definitely an *interpreter* for Ruby.\n\nBut what of CPython? When you run your Python program using it, the code is\nparsed and converted to an internal bytecode format, which is then executed\ninside the VM. From the user's perspective, this is clearly an interpreter --\nthey run their program from source. But if you look under CPython's scaly skin,\nyou'll see that there is definitely some compiling going on.\n\nThe answer is that it is both. CPython *is* an\ninterpreter, and it *has* a compiler. In practice, most scripting languages work\nthis way, as you can see:\n\n\n\n\"A\n\nThat overlapping region in the center is where our second interpreter lives too,\nsince it internally compiles to bytecode. So while this book is nominally about\ninterpreters, we'll cover some compilation too.\n\n## Our Journey\n\nThat's a lot to take in all at once. Don't worry. This isn't the chapter where\nyou're expected to *understand* all of these pieces and parts. I just want you\nto know that they are out there and roughly how they fit together.\n\nThis map should serve you well as you explore the territory beyond the guided\npath we take in this book. I want to leave you yearning to strike out on your\nown and wander all over that mountain.\n\nBut, for now, it's time for our own journey to begin. Tighten your bootlaces,\ncinch up your pack, and come along. From here on out,\nall you need to focus on is the path in front of you.\n\n\n\n
\n\n## Challenges\n\n1. Pick an open source implementation of a language you like. Download the\n source code and poke around in it. Try to find the code that implements the\n scanner and parser. Are they handwritten, or generated using tools like\n Lex and Yacc? (`.l` or `.y` files usually imply the latter.)\n\n1. Just-in-time compilation tends to be the fastest way to implement dynamically\n typed languages, but not all of them use it. What reasons are there to *not*\n JIT?\n\n1. Most Lisp implementations that compile to C also contain an interpreter that\n lets them execute Lisp code on the fly as well. Why?\n\n
\n" }, { "path": "book/a-tree-walk-interpreter.md", "content": "With this part, we begin jlox, the first of our two interpreters. Programming\nlanguages are a huge topic with piles of concepts and terminology to cram into\nyour brain all at once. Programming language theory requires a level of mental\nrigor that you probably haven't had to summon since your last calculus final.\n(Fortunately there isn't too much theory in this book.)\n\nImplementing an interpreter uses a few architectural tricks and design\npatterns uncommon in other kinds of applications, so we'll be getting used to\nthe engineering side of things too. Given all of that, we'll keep the code we\nhave to write as simple and plain as possible.\n\nIn less than two thousand lines of clean Java code, we'll build a complete\ninterpreter for Lox that implements every single feature of the language,\nexactly as we've specified. The first few chapters work front-to-back through\nthe phases of the interpreter -- [scanning][], [parsing][], and\n[evaluating code][]. After that, we add language features one at a time,\ngrowing a simple calculator into a full-fledged scripting language.\n\n[scanning]: scanning.html\n[parsing]: parsing-expressions.html\n[evaluating code]: evaluating-expressions.html\n" }, { "path": "book/a-virtual-machine.md", "content": "> Magicians protect their secrets not because the secrets are large and\n> important, but because they are so small and trivial. The wonderful effects\n> created on stage are often the result of a secret so absurd that the magician\n> would be embarrassed to admit that that was how it was done.\n>\n> Christopher Priest, The Prestige\n\nWe've spent a lot of time talking about how to represent a program as a sequence\nof bytecode instructions, but it feels like learning biology using only stuffed,\ndead animals. We know what instructions are in theory, but we've never seen them\nin action, so it's hard to really understand what they *do*. It would be hard to\nwrite a compiler that outputs bytecode when we don't have a good understanding\nof how that bytecode behaves.\n\nSo, before we go and build the front end of our new interpreter, we will begin\nwith the back end -- the virtual machine that executes instructions. It breathes\nlife into the bytecode. Watching the instructions prance around gives us a\nclearer picture of how a compiler might translate the user's source code into a\nseries of them.\n\n## An Instruction Execution Machine\n\nThe virtual machine is one part of our interpreter's internal architecture. You\nhand it a chunk of code -- literally a Chunk -- and it runs it. The code and\ndata structures for the VM reside in a new module.\n\n^code vm-h\n\nAs usual, we start simple. The VM will gradually acquire a whole pile of state\nit needs to keep track of, so we define a struct now to stuff that all in.\nCurrently, all we store is the chunk that it executes.\n\nLike we do with most of the data structures we create, we also define functions\nto create and tear down a VM. Here's the implementation:\n\n^code vm-c\n\nOK, calling those functions \"implementations\" is a stretch. We don't have any\ninteresting state to initialize or free yet, so the functions are empty. Trust\nme, we'll get there.\n\nThe slightly more interesting line here is that declaration of `vm`. This module\nis eventually going to have a slew of functions and it would be a chore to pass\naround a pointer to the VM to all of them. Instead, we declare a single global\nVM object. We need only one anyway, and this keeps the code in the book a little\nlighter on the page.\n\n\n\nBefore we start pumping fun code into our VM, let's go ahead and wire it up to\nthe interpreter's main entrypoint.\n\n^code main-init-vm (1 before, 1 after)\n\nWe spin up the VM when the interpreter first starts. Then when we're about to\nexit, we wind it down.\n\n^code main-free-vm (1 before, 1 after)\n\nOne last ceremonial obligation:\n\n^code main-include-vm (1 before, 2 after)\n\nNow when you run clox, it starts up the VM before it creates that hand-authored\nchunk from the [last chapter][]. The VM is ready and waiting, so let's teach it\nto do something.\n\n[last chapter]: chunks-of-bytecode.html#disassembling-chunks\n\n### Executing instructions\n\nThe VM springs into action when we command it to interpret a chunk of bytecode.\n\n^code main-interpret (1 before, 1 after)\n\nThis function is the main entrypoint into the VM. It's declared like so:\n\n^code interpret-h (1 before, 2 after)\n\nThe VM runs the chunk and then responds with a value from this enum:\n\n^code interpret-result (2 before, 2 after)\n\nWe aren't using the result yet, but when we have a compiler that reports static\nerrors and a VM that detects runtime errors, the interpreter will use this to\nknow how to set the exit code of the process.\n\nWe're inching towards some actual implementation.\n\n^code interpret\n\nFirst, we store the chunk being executed in the VM. Then we call `run()`, an\ninternal helper function that actually runs the bytecode instructions. Between\nthose two parts is an intriguing line. What is this `ip` business?\n\nAs the VM works its way through the bytecode, it keeps track of where it is --\nthe location of the instruction currently being executed. We don't use a local variable inside `run()` for this because eventually\nother functions will need to access it. Instead, we store it as a field in VM.\n\n\n\n^code ip (2 before, 1 after)\n\nIts type is a byte pointer. We use an actual real C pointer pointing right into\nthe middle of the bytecode array instead of something like an integer index\nbecause it's faster to dereference a pointer than look up an element in an array\nby index.\n\nThe name \"IP\" is traditional, and -- unlike many traditional names in CS --\nactually makes sense: it's an **[instruction pointer][ip]**. Almost every\ninstruction set in the world, real and virtual, has a\nregister or variable like this.\n\n[ip]: https://en.wikipedia.org/wiki/Program_counter\n\n\n\nWe initialize `ip` by pointing it at the first byte of code in the chunk. We\nhaven't executed that instruction yet, so `ip` points to the instruction *about\nto be executed*. This will be true during the entire time the VM is running: the\nIP always points to the next instruction, not the one currently being handled.\n\nThe real fun happens in `run`().\n\n^code run\n\nThis is the single most important function in all\nof clox, by far. When the interpreter executes a user's program, it will spend\nsomething like 90% of its time inside `run()`. It is the beating heart of the\nVM.\n\n\n\nDespite that dramatic intro, it's conceptually pretty simple. We have an outer\nloop that goes and goes. Each turn through that loop, we read and execute a\nsingle bytecode instruction.\n\nTo process an instruction, we first figure out what kind of instruction we're\ndealing with. The `READ_BYTE` macro reads the byte currently pointed at by `ip`\nand then advances the instruction pointer. The first\nbyte of any instruction is the opcode. Given a numeric opcode, we need to get to\nthe right C code that implements that instruction's semantics. This process is\ncalled **decoding** or **dispatching** the instruction.\n\n\n\nWe do that process for every single instruction, every single time one is\nexecuted, so this is the most performance critical part of the entire virtual\nmachine. Programming language lore is filled with clever techniques to do bytecode dispatch efficiently,\ngoing all the way back to the early days of computers.\n\n\n\nAlas, the fastest solutions require either non-standard extensions to C, or\nhandwritten assembly code. For clox, we'll keep it simple. Just like our\ndisassembler, we have a single giant `switch` statement with a case for each\nopcode. The body of each case implements that opcode's behavior.\n\nSo far, we handle only a single instruction, `OP_RETURN`, and the only thing it\ndoes is exit the loop entirely. Eventually, that instruction will be used to\nreturn from the current Lox function, but we don't have functions yet, so we'll\nrepurpose it temporarily to end the execution.\n\nLet's go ahead and support our one other instruction.\n\n^code op-constant (1 before, 1 after)\n\nWe don't have enough machinery in place yet to do anything useful with a\nconstant. For now, we'll just print it out so we interpreter hackers can see\nwhat's going on inside our VM. That call to `printf()` necessitates an include.\n\n^code vm-include-stdio (1 after)\n\nWe also have a new macro to define.\n\n^code read-constant (1 before, 2 after)\n\n`READ_CONSTANT()` reads the next byte from the bytecode, treats the resulting\nnumber as an index, and looks up the corresponding Value in the chunk's constant\ntable. In later chapters, we'll add a few more instructions with operands that\nrefer to constants, so we're setting up this helper macro now.\n\nLike the previous `READ_BYTE` macro, `READ_CONSTANT` is only used inside\n`run()`. To make that scoping more explicit, the macro definitions themselves\nare confined to that function. We define them at the\nbeginning and -- because we care -- undefine them at the end.\n\n^code undef-read-constant (1 before, 1 after)\n\n\n\n### Execution tracing\n\nIf you run clox now, it executes the chunk we hand-authored in the last chapter\nand spits out `1.2` to your terminal. We can see that it's working, but that's\nonly because our implementation of `OP_CONSTANT` has temporary code to log the\nvalue. Once that instruction is doing what it's supposed to do and plumbing that\nconstant along to other operations that want to consume it, the VM will become a\nblack box. That makes our lives as VM implementers harder.\n\nTo help ourselves out, now is a good time to add some diagnostic logging to the\nVM like we did with chunks themselves. In fact, we'll even reuse the same code.\nWe don't want this logging enabled all the time -- it's just for us VM hackers,\nnot Lox users -- so first we create a flag to hide it behind.\n\n^code define-debug-trace (1 before, 2 after)\n\nWhen this flag is defined, the VM disassembles and prints each instruction right\nbefore executing it. Where our previous disassembler walked an entire chunk\nonce, statically, this disassembles instructions dynamically, on the fly.\n\n^code trace-execution (1 before, 1 after)\n\nSince `disassembleInstruction()` takes an integer byte *offset* and we store the\ncurrent instruction reference as a direct pointer, we first do a little pointer\nmath to convert `ip` back to a relative offset from the beginning of the\nbytecode. Then we disassemble the instruction that begins at that byte.\n\nAs ever, we need to bring in the declaration of the function before we can call\nit.\n\n^code vm-include-debug (1 before, 1 after)\n\nI know this code isn't super impressive so far -- it's literally a switch\nstatement wrapped in a `for` loop but, believe it or not, this is one of the two\nmajor components of our VM. With this, we can imperatively execute instructions.\nIts simplicity is a virtue -- the less work it does, the faster it can do it.\nContrast this with all of the complexity and overhead we had in jlox with the\nVisitor pattern for walking the AST.\n\n## A Value Stack Manipulator\n\nIn addition to imperative side effects, Lox has expressions that produce,\nmodify, and consume values. Thus, our compiled bytecode needs a way to shuttle\nvalues around between the different instructions that need them. For example:\n\n```lox\nprint 3 - 2;\n```\n\nWe obviously need instructions for the constants 3 and 2, the `print` statement,\nand the subtraction. But how does the subtraction instruction know that 3 is\nthe minuend and 2 is the subtrahend? How does the print\ninstruction know to print the result of that?\n\n\n\nTo put a finer point on it, look at this thing right here:\n\n```lox\nfun echo(n) {\n print n;\n return n;\n}\n\nprint echo(echo(1) + echo(2)) + echo(echo(4) + echo(5));\n```\n\nI wrapped each subexpression in a call to `echo()` that prints and returns its\nargument. That side effect means we can see the exact order of operations.\n\nDon't worry about the VM for a minute. Think about just the semantics of Lox\nitself. The operands to an arithmetic operator obviously need to be evaluated\nbefore we can perform the operation itself. (It's pretty hard to add `a + b` if\nyou don't know what `a` and `b` are.) Also, when we implemented expressions in\njlox, we decided that the left operand must be\nevaluated before the right.\n\n\n\nHere is the syntax tree for the `print` statement:\n\n\"The\n\nGiven left-to-right evaluation, and the way the expressions are nested, any\ncorrect Lox implementation *must* print these numbers in this order:\n\n```text\n1 // from echo(1)\n2 // from echo(2)\n3 // from echo(1 + 2)\n4 // from echo(4)\n5 // from echo(5)\n9 // from echo(4 + 5)\n12 // from print 3 + 9\n```\n\nOur old jlox interpreter accomplishes this by recursively traversing the AST. It\ndoes a postorder traversal. First it recurses down the left operand branch,\nthen the right operand, then finally it evaluates the node itself.\n\nAfter evaluating the left operand, jlox needs to store that result somewhere\ntemporarily while it's busy traversing down through the right operand tree. We\nuse a local variable in Java for that. Our recursive tree-walk interpreter\ncreates a unique Java call frame for each node being evaluated, so we could have\nas many of these local variables as we needed.\n\nIn clox, our `run()` function is not recursive -- the nested expression tree is\nflattened out into a linear series of instructions. We don't have the luxury of\nusing C local variables, so how and where should we store these temporary\nvalues? You can probably guess already, but I want to\nreally drill into this because it's an aspect of programming that we take for\ngranted, but we rarely learn *why* computers are architected this way.\n\n\n\nLet's do a weird exercise. We'll walk through the execution of the above program\na step at a time:\n\n\"The\n\nOn the left are the steps of code. On the right are the values we're tracking.\nEach bar represents a number. It starts when the value is first produced --\neither a constant or the result of an addition. The length of the bar tracks\nwhen a previously produced value needs to be kept around, and it ends when that\nvalue finally gets consumed by an operation.\n\nAs you step through, you see values appear and then later get eaten. The\nlongest-lived ones are the values produced from the left-hand side of an\naddition. Those stick around while we work through the right-hand operand\nexpression.\n\nIn the above diagram, I gave each unique number its own visual column. Let's be\na little more parsimonious. Once a number is consumed, we allow its column to be\nreused for another later value. In other words, we take all of those gaps\nup there and fill them in, pushing in numbers from the right:\n\n\"Like\n\nThere's some interesting stuff going on here. When we shift everything over,\neach number still manages to stay in a single column for its entire life. Also,\nthere are no gaps left. In other words, whenever a number appears earlier than\nanother, then it will live at least as long as that second one. The first number\nto appear is the last to be consumed. Hmm... last-in, first-out... why, that's a\nstack!\n\n\n\nIn the second diagram, each time we introduce a number, we push it onto the\nstack from the right. When numbers are consumed, they are always popped off from\nrightmost to left.\n\nSince the temporary values we need to track naturally have stack-like behavior,\nour VM will use a stack to manage them. When an instruction \"produces\" a value,\nit pushes it onto the stack. When it needs to consume one or more values, it\ngets them by popping them off the stack.\n\n### The VM's Stack\n\nMaybe this doesn't seem like a revelation, but I *love* stack-based VMs. When\nyou first see a magic trick, it feels like something actually magical. But then\nyou learn how it works -- usually some mechanical gimmick or misdirection -- and\nthe sense of wonder evaporates. There are a couple of\nideas in computer science where even after I pulled them apart and learned all\nthe ins and outs, some of the initial sparkle remained. Stack-based VMs are one\nof those.\n\n\n\nAs you'll see in this chapter, executing instructions in a stack-based VM is\ndead simple. In later chapters, you'll also discover\nthat compiling a source language to a stack-based instruction set is a piece of\ncake. And yet, this architecture is fast enough to be used by production\nlanguage implementations. It almost feels like cheating at the programming\nlanguage game.\n\n\n\nAlrighty, it's codin' time! Here's the stack:\n\n^code vm-stack (3 before, 1 after)\n\nWe implement the stack semantics ourselves on top of a raw C array. The bottom\nof the stack -- the first value pushed and the last to be popped -- is at\nelement zero in the array, and later pushed values follow it. If we push the\nletters of \"crepe\" -- my favorite stackable breakfast item -- onto the stack, in\norder, the resulting C array looks like this:\n\n\"An\n\nSince the stack grows and shrinks as values are pushed and popped, we need to\ntrack where the top of the stack is in the array. As with `ip`, we use a direct\npointer instead of an integer index since it's faster to dereference the pointer\nthan calculate the offset from the index each time we need it.\n\nThe pointer points at the array element just *past* the element containing the\ntop value on the stack. That seems a little odd, but almost every implementation\ndoes this. It means we can indicate that the stack is empty by pointing at\nelement zero in the array.\n\n\"An\n\nIf we pointed to the top element, then for an empty stack we'd need to point at\nelement -1. That's undefined in C. As we push values\nonto the stack...\n\n\n\n\"An\n\n...`stackTop` always points just past the last item.\n\n\"An\n\nI remember it like this: `stackTop` points to where the next value to be pushed\nwill go. The maximum number of values we can store on the stack (for now, at\nleast) is:\n\n^code stack-max (1 before, 2 after)\n\nGiving our VM a fixed stack size means it's possible for some sequence of\ninstructions to push too many values and run out of stack space -- the classic\n\"stack overflow\". We could grow the stack dynamically as needed, but for now\nwe'll keep it simple. Since VM uses Value, we need to include its declaration.\n\n^code vm-include-value (1 before, 2 after)\n\nNow that VM has some interesting state, we get to initialize it.\n\n^code call-reset-stack (1 before, 1 after)\n\nThat uses this helper function:\n\n^code reset-stack\n\nSince the stack array is declared directly inline in the VM struct, we don't\nneed to allocate it. We don't even need to clear the unused cells in the\narray -- we simply won't access them until after values have been stored in\nthem. The only initialization we need is to set `stackTop` to point to the\nbeginning of the array to indicate that the stack is empty.\n\nThe stack protocol supports two operations:\n\n^code push-pop (1 before, 2 after)\n\nYou can push a new value onto the top of the stack, and you can pop the most\nrecently pushed value back off. Here's the first function:\n\n^code push\n\nIf you're rusty on your C pointer syntax and operations, this is a good warm-up.\nThe first line stores `value` in the array element at the top of the stack.\nRemember, `stackTop` points just *past* the last used element, at the next\navailable one. This stores the value in that slot. Then we increment the pointer\nitself to point to the next unused slot in the array now that the previous slot\nis occupied.\n\nPopping is the mirror image.\n\n^code pop\n\nFirst, we move the stack pointer *back* to get to the most recent used slot in\nthe array. Then we look up the value at that index and return it. We don't need\nto explicitly \"remove\" it from the array -- moving `stackTop` down is enough to\nmark that slot as no longer in use.\n\n### Stack tracing\n\nWe have a working stack, but it's hard to *see* that it's working. When we start\nimplementing more complex instructions and compiling and running larger pieces\nof code, we'll end up with a lot of values crammed into that array. It would\nmake our lives as VM hackers easier if we had some visibility into the stack.\n\nTo that end, whenever we're tracing execution, we'll also show the current\ncontents of the stack before we interpret each instruction.\n\n^code trace-stack (1 before, 1 after)\n\nWe loop, printing each value in the array, starting at the first (bottom of the\nstack) and ending when we reach the top. This lets us observe the effect of each\ninstruction on the stack. The output is pretty verbose, but it's useful when\nwe're surgically extracting a nasty bug from the bowels of the interpreter.\n\nStack in hand, let's revisit our two instructions. First up:\n\n^code push-constant (2 before, 1 after)\n\nIn the last chapter, I was hand-wavey about how the `OP_CONSTANT` instruction\n\"loads\" a constant. Now that we have a stack you know what it means to actually\nproduce a value: it gets pushed onto the stack.\n\n^code print-return (1 before, 1 after)\n\nThen we make `OP_RETURN` pop the stack and print the top value before exiting.\nWhen we add support for real functions to clox, we'll change this code. But, for\nnow, it gives us a way to get the VM executing simple instruction sequences and\ndisplaying the result.\n\n## An Arithmetic Calculator\n\nThe heart and soul of our VM are in place now. The bytecode loop dispatches and\nexecutes instructions. The stack grows and shrinks as values flow through it.\nThe two halves work, but it's hard to get a feel for how cleverly they interact\nwith only the two rudimentary instructions we have so far. So let's teach our\ninterpreter to do arithmetic.\n\nWe'll start with the simplest arithmetic operation, unary negation.\n\n```lox\nvar a = 1.2;\nprint -a; // -1.2.\n```\n\nThe prefix `-` operator takes one operand, the value to negate. It produces a\nsingle result. We aren't fussing with a parser yet, but we can add the\nbytecode instruction that the above syntax will compile to.\n\n^code negate-op (1 before, 1 after)\n\nWe execute it like so:\n\n^code op-negate (1 before, 1 after)\n\nThe instruction needs a value to operate on, which it gets by popping from the\nstack. It negates that, then pushes the result back on for later instructions to\nuse. Doesn't get much easier than that. We can disassemble it too.\n\n^code disassemble-negate (2 before, 1 after)\n\nAnd we can try it out in our test chunk.\n\n^code main-negate (1 before, 2 after)\n\nAfter loading the constant, but before returning, we execute the negate\ninstruction. That replaces the constant on the stack with its negation. Then the\nreturn instruction prints that out:\n\n```text\n-1.2\n```\n\nMagical!\n\n### Binary operators\n\nOK, unary operators aren't *that* impressive. We still only ever have a single\nvalue on the stack. To really see some depth, we need binary operators. Lox has\nfour binary arithmetic operators: addition, subtraction,\nmultiplication, and division. We'll go ahead and implement them all at the same\ntime.\n\n\n\n^code binary-ops (1 before, 1 after)\n\nBack in the bytecode loop, they are executed like this:\n\n^code op-binary (1 before, 1 after)\n\nThe only difference between these four instructions is which underlying C\noperator they ultimately use to combine the two operands. Surrounding that core\narithmetic expression is some boilerplate code to pull values off the stack and\npush the result. When we later add dynamic typing, that boilerplate will grow.\nTo avoid repeating that code four times, I wrapped it up in a macro.\n\n^code binary-op (1 before, 2 after)\n\nI admit this is a fairly adventurous use of the C\npreprocessor. I hesitated to do this, but you'll be glad in later chapters when\nwe need to add the type checking for each operand and stuff. It would be a chore\nto walk you through the same code four times.\n\n\n\nIf you aren't familiar with the trick already, that outer `do while` loop\nprobably looks really weird. This macro needs to expand to a series of\nstatements. To be careful macro authors, we want to ensure those statements all\nend up in the same scope when the macro is expanded. Imagine if you defined:\n\n```c\n#define WAKE_UP() makeCoffee(); drinkCoffee();\n```\n\nAnd then used it like:\n\n```c\nif (morning) WAKE_UP();\n```\n\nThe intent is to execute both statements of the macro body only if `morning` is\ntrue. But it expands to:\n\n```c\nif (morning) makeCoffee(); drinkCoffee();;\n```\n\nOops. The `if` attaches only to the *first* statement. You might think you could\nfix this using a block.\n\n```c\n#define WAKE_UP() { makeCoffee(); drinkCoffee(); }\n```\n\nThat's better, but you still risk:\n\n```c\nif (morning)\n WAKE_UP();\nelse\n sleepIn();\n```\n\nNow you get a compile error on the `else` because of that trailing `;` after the\nmacro's block. Using a `do while` loop in the macro looks funny, but it gives\nyou a way to contain multiple statements inside a block that *also* permits a\nsemicolon at the end.\n\nWhere were we? Right, so what the body of that macro does is straightforward. A\nbinary operator takes two operands, so it pops twice. It performs the operation\non those two values and then pushes the result.\n\nPay close attention to the *order* of the two pops. Note that we assign the\nfirst popped operand to `b`, not `a`. It looks backwards. When the operands\nthemselves are calculated, the left is evaluated first, then the right. That\nmeans the left operand gets pushed before the right operand. So the right\noperand will be on top of the stack. Thus, the first value we pop is `b`.\n\nFor example, if we compile `3 - 1`, the data flow between the instructions looks\nlike so:\n\n\"A\n\nAs we did with the other macros inside `run()`, we clean up after ourselves at\nthe end of the function.\n\n^code undef-binary-op (1 before, 1 after)\n\nLast is disassembler support.\n\n^code disassemble-binary (2 before, 1 after)\n\nThe arithmetic instruction formats are simple, like `OP_RETURN`. Even though the\narithmetic *operators* take operands -- which are found on the stack -- the\narithmetic *bytecode instructions* do not.\n\nLet's put some of our new instructions through their paces by evaluating a\nlarger expression:\n\n\"The\n\nBuilding on our existing example chunk, here's the additional instructions we\nneed to hand-compile that AST to bytecode.\n\n^code main-chunk (3 before, 3 after)\n\nThe addition goes first. The instruction for the left constant, 1.2, is already\nthere, so we add another for 3.4. Then we add those two using `OP_ADD`, leaving\nit on the stack. That covers the left side of the division. Next we push the\n5.6, and divide the result of the addition by it. Finally, we negate the result\nof that.\n\nNote how the output of the `OP_ADD` implicitly flows into being an operand of\n`OP_DIVIDE` without either instruction being directly coupled to each other.\nThat's the magic of the stack. It lets us freely compose instructions without\nthem needing any complexity or awareness of the data flow. The stack acts like a\nshared workspace that they all read from and write to.\n\nIn this tiny example chunk, the stack still only gets two values tall, but when\nwe start compiling Lox source to bytecode, we'll have chunks that use much more\nof the stack. In the meantime, try playing around with this hand-authored chunk\nto calculate different nested arithmetic expressions and see how values flow\nthrough the instructions and stack.\n\nYou may as well get it out of your system now. This is the last chunk we'll\nbuild by hand. When we next revisit bytecode, we will be writing a compiler to\ngenerate it for us.\n\n
\n\n## Challenges\n\n1. What bytecode instruction sequences would you generate for the following\n expressions:\n\n ```lox\n 1 * 2 + 3\n 1 + 2 * 3\n 3 - 2 - 1\n 1 + 2 * 3 - 4 / -5\n ```\n\n (Remember that Lox does not have a syntax for negative number literals, so\n the `-5` is negating the number 5.)\n\n1. If we really wanted a minimal instruction set, we could eliminate either\n `OP_NEGATE` or `OP_SUBTRACT`. Show the bytecode instruction sequence you\n would generate for:\n\n ```lox\n 4 - 3 * -2\n ```\n\n First, without using `OP_NEGATE`. Then, without using `OP_SUBTRACT`.\n\n Given the above, do you think it makes sense to have both instructions? Why\n or why not? Are there any other redundant instructions you would consider\n including?\n\n1. Our VM's stack has a fixed size, and we don't check if pushing a value\n overflows it. This means the wrong series of instructions could cause our\n interpreter to crash or go into undefined behavior. Avoid that by\n dynamically growing the stack as needed.\n\n What are the costs and benefits of doing so?\n\n1. To interpret `OP_NEGATE`, we pop the operand, negate the value, and then\n push the result. That's a simple implementation, but it increments and\n decrements `stackTop` unnecessarily, since the stack ends up the same height\n in the end. It might be faster to simply negate the value in place on the\n stack and leave `stackTop` alone. Try that and see if you can measure a\n performance difference.\n\n Are there other instructions where you can do a similar optimization?\n\n
\n\n
\n\n## Design Note: Register-Based Bytecode\n\nFor the remainder of this book, we'll meticulously implement an interpreter\naround a stack-based bytecode instruction set. There's another family of\nbytecode architectures out there -- *register-based*. Despite the name, these\nbytecode instructions aren't quite as difficult to work with as the registers in\nan actual chip like x64. With real hardware registers,\nyou usually have only a handful for the entire program, so you spend a lot of\neffort [trying to use them efficiently and shuttling stuff in and out of\nthem][register allocation].\n\n[register allocation]: https://en.wikipedia.org/wiki/Register_allocation\n\n\n\nIn a register-based VM, you still have a stack. Temporary values still get\npushed onto it and popped when no longer needed. The main difference is that\ninstructions can read their inputs from anywhere in the stack and can store\ntheir outputs into specific stack slots.\n\nTake this little Lox script:\n\n```lox\nvar a = 1;\nvar b = 2;\nvar c = a + b;\n```\n\nIn our stack-based VM, the last statement will get compiled to something like:\n\n```lox\nload // Read local variable a and push onto stack.\nload // Read local variable b and push onto stack.\nadd // Pop two values, add, push result.\nstore // Pop value and store in local variable c.\n```\n\n(Don't worry if you don't fully understand the load and store instructions yet.\nWe'll go over them in much greater detail [when we implement\nvariables][variables].) We have four separate instructions. That means four\ntimes through the bytecode interpret loop, four instructions to decode and\ndispatch. It's at least seven bytes of code -- four for the opcodes and another\nthree for the operands identifying which locals to load and store. Three pushes\nand three pops. A lot of work!\n\n[variables]: global-variables.html\n\nIn a register-based instruction set, instructions can read from and store\ndirectly into local variables. The bytecode for the last statement above looks\nlike:\n\n```lox\nadd // Read values from a and b, add, store in c.\n```\n\nThe add instruction is bigger -- it has three instruction operands that define\nwhere in the stack it reads its inputs from and writes the result to. But since\nlocal variables live on the stack, it can read directly from `a` and `b` and\nthen store the result right into `c`.\n\nThere's only a single instruction to decode and dispatch, and the whole thing\nfits in four bytes. Decoding is more complex because of the additional operands,\nbut it's still a net win. There's no pushing and popping or other stack\nmanipulation.\n\nThe main implementation of Lua used to be stack-based. For Lua\n5.0, the implementers switched to a register instruction set and noted a\nspeed improvement. The amount of improvement, naturally, depends heavily on the\ndetails of the language semantics, specific instruction set, and compiler\nsophistication, but that should get your attention.\n\n\n\nThat raises the obvious question of why I'm going to spend the rest of the book\ndoing a stack-based bytecode. Register VMs are neat, but they are quite a bit\nharder to write a compiler for. For what is likely to be your very first\ncompiler, I wanted to stick with an instruction set that's easy to generate and\neasy to execute. Stack-based bytecode is marvelously simple.\n\nIt's also *much* better known in the literature and the community. Even though\nyou may eventually move to something more advanced, it's a good common ground to\nshare with the rest of your language hacker peers.\n\n
\n" }, { "path": "book/acknowledgements.md", "content": "When the first copy of \"[Game Programming Patterns][gpp]\" sold, I guess I had\nthe right to call myself an author. But it took time to feel comfortable with\nthat label. Thank you to everyone who bought copies of my first book, and to the\npublishers and translators who brought it to other languages. You gave me the\nconfidence to believe I could tackle a project of this scope. Well, that, and\nmassively underestimating what I was getting myself into, but that's on me.\n\n[gpp]: https://gameprogrammingpatterns.com/\n\nA fear particular to technical writing is *getting stuff wrong*. Tests and\nstatic analysis only get you so far. Once the code and prose is in ink on paper,\nthere's no fixing it. I am deeply grateful to the many people who filed issues\nand pull requests on the [open source repo][repo] for the book. Special thanks\ngo to cm1776, who filed 145 tactfully worded issues pointing out hundreds of\ncode errors, typos, and unclear sentences. The book is more accurate and\nreadable because of you all.\n\n[repo]: https://github.com/munificent/craftinginterpreters\n\nI'm grateful to my copy editor Kari Somerton who braved a heap of computer\nscience jargon and an unfamilar workflow in order to fix my many grammar errors\nand stylistic inconsistencies.\n\nWhen the pandemic turned everyone's life upside down, a number of people reached\nout to tell me that my book provided a helpful distraction. This book that I\nspent six years writing forms a chapter in my own life's story and I'm grateful\nto the readers who contacted me and made that chapter more meaningful.\n\nFinally, the deepest thanks go to my wife Megan and my daughters Lily and\nGretchen. You patiently endured the time I had to sink into the book, and my\nstress while writing it. There's no one I'd rather be stuck at home with.\n" }, { "path": "book/appendix-i.md", "content": "Here is a complete grammar for Lox. The chapters that introduce each part of the\nlanguage include the grammar rules there, but this collects them all into one\nplace.\n\n## Syntax Grammar\n\nThe syntactic grammar is used to parse the linear sequence of tokens into the\nnested syntax tree structure. It starts with the first rule that matches an\nentire Lox program (or a single REPL entry).\n\n```ebnf\nprogram → declaration* EOF ;\n```\n\n### Declarations\n\nA program is a series of declarations, which are the statements that bind new\nidentifiers or any of the other statement types.\n\n```ebnf\ndeclaration → classDecl\n | funDecl\n | varDecl\n | statement ;\n\nclassDecl → \"class\" IDENTIFIER ( \"<\" IDENTIFIER )?\n \"{\" function* \"}\" ;\nfunDecl → \"fun\" function ;\nvarDecl → \"var\" IDENTIFIER ( \"=\" expression )? \";\" ;\n```\n\n### Statements\n\nThe remaining statement rules produce side effects, but do not introduce\nbindings.\n\n```ebnf\nstatement → exprStmt\n | forStmt\n | ifStmt\n | printStmt\n | returnStmt\n | whileStmt\n | block ;\n\nexprStmt → expression \";\" ;\nforStmt → \"for\" \"(\" ( varDecl | exprStmt | \";\" )\n expression? \";\"\n expression? \")\" statement ;\nifStmt → \"if\" \"(\" expression \")\" statement\n ( \"else\" statement )? ;\nprintStmt → \"print\" expression \";\" ;\nreturnStmt → \"return\" expression? \";\" ;\nwhileStmt → \"while\" \"(\" expression \")\" statement ;\nblock → \"{\" declaration* \"}\" ;\n```\n\nNote that `block` is a statement rule, but is also used as a nonterminal in a\ncouple of other rules for things like function bodies.\n\n### Expressions\n\nExpressions produce values. Lox has a number of unary and binary operators with\ndifferent levels of precedence. Some grammars for languages do not directly\nencode the precedence relationships and specify that elsewhere. Here, we use a\nseparate rule for each precedence level to make it explicit.\n\n```ebnf\nexpression → assignment ;\n\nassignment → ( call \".\" )? IDENTIFIER \"=\" assignment\n | logic_or ;\n\nlogic_or → logic_and ( \"or\" logic_and )* ;\nlogic_and → equality ( \"and\" equality )* ;\nequality → comparison ( ( \"!=\" | \"==\" ) comparison )* ;\ncomparison → term ( ( \">\" | \">=\" | \"<\" | \"<=\" ) term )* ;\nterm → factor ( ( \"-\" | \"+\" ) factor )* ;\nfactor → unary ( ( \"/\" | \"*\" ) unary )* ;\n\nunary → ( \"!\" | \"-\" ) unary | call ;\ncall → primary ( \"(\" arguments? \")\" | \".\" IDENTIFIER )* ;\nprimary → \"true\" | \"false\" | \"nil\" | \"this\"\n | NUMBER | STRING | IDENTIFIER | \"(\" expression \")\"\n | \"super\" \".\" IDENTIFIER ;\n```\n\n### Utility rules\n\nIn order to keep the above rules a little cleaner, some of the grammar is\nsplit out into a few reused helper rules.\n\n```ebnf\nfunction → IDENTIFIER \"(\" parameters? \")\" block ;\nparameters → IDENTIFIER ( \",\" IDENTIFIER )* ;\narguments → expression ( \",\" expression )* ;\n```\n\n## Lexical Grammar\n\nThe lexical grammar is used by the scanner to group characters into tokens.\nWhere the syntax is [context free][], the lexical grammar is [regular][] -- note\nthat there are no recursive rules.\n\n[context free]: https://en.wikipedia.org/wiki/Context-free_grammar\n[regular]: https://en.wikipedia.org/wiki/Regular_grammar\n\n```ebnf\nNUMBER → DIGIT+ ( \".\" DIGIT+ )? ;\nSTRING → \"\\\"\" * \"\\\"\" ;\nIDENTIFIER → ALPHA ( ALPHA | DIGIT )* ;\nALPHA → \"a\" ... \"z\" | \"A\" ... \"Z\" | \"_\" ;\nDIGIT → \"0\" ... \"9\" ;\n```\n" }, { "path": "book/appendix-ii.md", "content": "For your edification, here is the code produced by [the little script\nwe built][generator] to automate generating the syntax tree classes for jlox.\n\n[generator]: representing-code.html#metaprogramming-the-trees\n\n## Expressions\n\nExpressions are the first syntax tree nodes we see, introduced in \"[Representing\nCode](representing-code.html)\". The main Expr class defines the visitor\ninterface used to dispatch against the specific expression types, and contains\nthe other expression subclasses as nested classes.\n\n^code expr\n\n### Assign expression\n\nVariable assignment is introduced in \"[Statements and\nState](statements-and-state.html#assignment)\".\n\n^code expr-assign\n\n### Binary expression\n\nBinary operators are introduced in \"[Representing\nCode](representing-code.html)\".\n\n^code expr-binary\n\n### Call expression\n\nFunction call expressions are introduced in\n\"[Functions](functions.html#function-calls)\".\n\n^code expr-call\n\n### Get expression\n\nProperty access, or \"get\" expressions are introduced in\n\"[Classes](classes.html#properties-on-instances)\".\n\n^code expr-get\n\n### Grouping expression\n\nUsing parentheses to group expressions is introduced in \"[Representing\nCode](representing-code.html)\".\n\n^code expr-grouping\n\n### Literal expression\n\nLiteral value expressions are introduced in \"[Representing\nCode](representing-code.html)\".\n\n^code expr-literal\n\n### Logical expression\n\nThe logical `and` and `or` operators are introduced in \"[Control\nFlow](control-flow.html#logical-operators)\".\n\n^code expr-logical\n\n### Set expression\n\nProperty assignment, or \"set\" expressions are introduced in\n\"[Classes](classes.html#properties-on-instances)\".\n\n^code expr-set\n\n### Super expression\n\nThe `super` expression is introduced in\n\"[Inheritance](inheritance.html#calling-superclass-methods)\".\n\n^code expr-super\n\n### This expression\n\nThe `this` expression is introduced in \"[Classes](classes.html#this)\".\n\n^code expr-this\n\n### Unary expression\n\nUnary operators are introduced in \"[Representing Code](representing-code.html)\".\n\n^code expr-unary\n\n### Variable expression\n\nVariable access expressions are introduced in \"[Statements and\nState](statements-and-state.html#variable-syntax)\".\n\n^code expr-variable\n\n## Statements\n\nStatements form a second hierarchy of syntax tree nodes independent of\nexpressions. We add the first couple of them in \"[Statements and\nState](statements-and-state.html)\".\n\n^code stmt\n\n### Block statement\n\nThe curly-braced block statement that defines a local scope is introduced in\n\"[Statements and State](statements-and-state.html#block-syntax-and-semantics)\".\n\n^code stmt-block\n\n### Class statement\n\nClass declarations are introduced in, unsurprisingly,\n\"[Classes](classes.html#class-declarations)\".\n\n^code stmt-class\n\n### Expression statement\n\nThe expression statement is introduced in \"[Statements and\nState](statements-and-state.html#statements)\".\n\n^code stmt-expression\n\n### Function statement\n\nFunction declarations are introduced in, you guessed it,\n\"[Functions](functions.html#function-declarations)\".\n\n^code stmt-function\n\n### If statement\n\nThe `if` statement is introduced in \"[Control\nFlow](control-flow.html#conditional-execution)\".\n\n^code stmt-if\n\n### Print statement\n\nThe `print` statement is introduced in \"[Statements and\nState](statements-and-state.html#statements)\".\n\n^code stmt-print\n\n### Return statement\n\nYou need a function to return from, so `return` statements are introduced in\n\"[Functions](functions.html#return-statements)\".\n\n^code stmt-return\n\n### Variable statement\n\nVariable declarations are introduced in \"[Statements and\nState](statements-and-state.html#variable-syntax)\".\n\n^code stmt-var\n\n### While statement\n\nThe `while` statement is introduced in \"[Control\nFlow](control-flow.html#while-loops)\".\n\n^code stmt-while\n" }, { "path": "book/backmatter.md", "content": "You've reached the end of the book! There are two pieces of supplementary\nmaterial you may find helpful:\n\n* **[Appendix I][]** contains a complete grammar for Lox, all in one place.\n\n* **[Appendix II][]** shows the Java classes produced by [the AST generator][]\n we use for jlox.\n\n[appendix i]: appendix-i.html\n[appendix ii]: appendix-ii.html\n[the ast generator]: representing-code.html#metaprogramming-the-trees\n" }, { "path": "book/calls-and-functions.md", "content": "> Any problem in computer science can be solved with another level of\n> indirection. Except for the problem of too many layers of indirection.\n>\n> David Wheeler\n\nThis chapter is a beast. I try to break features into bite-sized pieces, but\nsometimes you gotta swallow the whole meal. Our next\ntask is functions. We could start with only function declarations, but that's\nnot very useful when you can't call them. We could do calls, but there's nothing\nto call. And all of the runtime support needed in the VM to support both of\nthose isn't very rewarding if it isn't hooked up to anything you can see. So\nwe're going to do it all. It's a lot, but we'll feel good when we're done.\n\n\n\n## Function Objects\n\nThe most interesting structural change in the VM is around the stack. We already\n*have* a stack for local variables and temporaries, so we're partway there. But\nwe have no notion of a *call* stack. Before we can make much progress, we'll\nhave to fix that. But first, let's write some code. I always feel better once I\nstart moving. We can't do much without having some kind of representation for\nfunctions, so we'll start there. From the VM's perspective, what is a function?\n\nA function has a body that can be executed, so that means some bytecode. We\ncould compile the entire program and all of its function declarations into one\nbig monolithic Chunk. Each function would have a pointer to the first\ninstruction of its code inside the Chunk.\n\nThis is roughly how compilation to native code works where you end up with one\nsolid blob of machine code. But for our bytecode VM, we can do something a\nlittle higher level. I think a cleaner model is to give each function its own\nChunk. We'll want some other metadata too, so let's go ahead and stuff it all in\na struct now.\n\n^code obj-function (2 before, 2 after)\n\nFunctions are first class in Lox, so they need to be actual Lox objects. Thus\nObjFunction has the same Obj header that all object types share. The `arity`\nfield stores the number of parameters the function expects. Then, in addition to\nthe chunk, we store the function's name. That will be\nhandy for reporting readable runtime errors.\n\n\n\nThis is the first time the \"object\" module has needed to reference Chunk, so we\nget an include.\n\n^code object-include-chunk (1 before, 1 after)\n\nLike we did with strings, we define some accessories to make Lox functions\neasier to work with in C. Sort of a poor man's object orientation. First, we'll\ndeclare a C function to create a new Lox function.\n\n^code new-function-h (3 before, 1 after)\n\nThe implementation is over here:\n\n^code new-function\n\nWe use our friend `ALLOCATE_OBJ()` to allocate memory and initialize the\nobject's header so that the VM knows what type of object it is. Instead of\npassing in arguments to initialize the function like we did with ObjString, we\nset the function up in a sort of blank state -- zero arity, no name, and no\ncode. That will get filled in later after the function is created.\n\nSince we have a new kind of object, we need a new object type in the enum.\n\n^code obj-type-function (1 before, 2 after)\n\nWhen we're done with a function object, we must return the bits it borrowed back\nto the operating system.\n\n^code free-function (1 before, 1 after)\n\nThis switch case is responsible for freeing the\nObjFunction itself as well as any other memory it owns. Functions own their\nchunk, so we call Chunk's destructor-like function.\n\n\n\nLox lets you print any object, and functions are first-class objects, so we\nneed to handle them too.\n\n^code print-function (1 before, 1 after)\n\nThis calls out to:\n\n^code print-function-helper\n\nSince a function knows its name, it may as well say it.\n\nFinally, we have a couple of macros for converting values to functions. First,\nmake sure your value actually *is* a function.\n\n^code is-function (2 before, 1 after)\n\nAssuming that evaluates to true, you can then safely cast the Value to an\nObjFunction pointer using this:\n\n^code as-function (2 before, 1 after)\n\nWith that, our object model knows how to represent functions. I'm feeling warmed\nup now. You ready for something a little harder?\n\n## Compiling to Function Objects\n\nRight now, our compiler assumes it is always compiling to one single chunk. With\neach function's code living in separate chunks, that gets more complex. When the\ncompiler reaches a function declaration, it needs to emit code into the\nfunction's chunk when compiling its body. At the end of the function body, the\ncompiler needs to return to the previous chunk it was working with.\n\nThat's fine for code inside function bodies, but what about code that isn't? The\n\"top level\" of a Lox program is also imperative code and we need a chunk to\ncompile that into. We can simplify the compiler and VM by placing that top-level\ncode inside an automatically defined function too. That way, the compiler is\nalways within some kind of function body, and the VM always runs code by\ninvoking a function. It's as if the entire program is wrapped inside an implicit `main()` function.\n\n\n\nBefore we get to user-defined functions, then, let's do the reorganization to\nsupport that implicit top-level function. It starts with the Compiler struct.\nInstead of pointing directly to a Chunk that the compiler writes to, it instead\nhas a reference to the function object being built.\n\n^code function-fields (1 before, 1 after)\n\nWe also have a little FunctionType enum. This lets the compiler tell when it's\ncompiling top-level code versus the body of a function. Most of the compiler\ndoesn't care about this -- that's why it's a useful abstraction -- but in one or\ntwo places the distinction is meaningful. We'll get to one later.\n\n^code function-type-enum\n\nEvery place in the compiler that was writing to the Chunk now needs to go\nthrough that `function` pointer. Fortunately, many chapters ago, we encapsulated access to the chunk in the\n`currentChunk()` function. We only need to fix that and the rest of the compiler\nis happy.\n\n\n\n^code current-chunk (1 before, 2 after)\n\nThe current chunk is always the chunk owned by the function we're in the middle\nof compiling. Next, we need to actually create that function. Previously, the VM\npassed a Chunk to the compiler which filled it with code. Instead, the compiler\nwill create and return a function that contains the compiled top-level code --\nwhich is all we support right now -- of the user's program.\n\n### Creating functions at compile time\n\nWe start threading this through in `compile()`, which is the main entry point\ninto the compiler.\n\n^code call-init-compiler (1 before, 2 after)\n\nThere are a bunch of changes in how the compiler is initialized. First, we\ninitialize the new Compiler fields.\n\n^code init-compiler (1 after)\n\nThen we allocate a new function object to compile into.\n\n^code init-function (1 before, 1 after)\n\n\n\n\n\nCreating an ObjFunction in the compiler might seem a little strange. A function\nobject is the *runtime* representation of a function, but here we are creating\nit at compile time. The way to think of it is that a function is similar to a\nstring or number literal. It forms a bridge between the compile time and runtime\nworlds. When we get to function *declarations*, those really *are* literals\n-- they are a notation that produces values of a built-in type. So the compiler creates function objects during compilation.\nThen, at runtime, they are simply invoked.\n\n\n\nHere is another strange piece of code:\n\n^code init-function-slot (1 before, 1 after)\n\nRemember that the compiler's `locals` array keeps track of which stack slots are\nassociated with which local variables or temporaries. From now on, the compiler\nimplicitly claims stack slot zero for the VM's own internal use. We give it an\nempty name so that the user can't write an identifier that refers to it. I'll\nexplain what this is about when it becomes useful.\n\nThat's the initialization side. We also need a couple of changes on the other\nend when we finish compiling some code.\n\n^code end-compiler (1 after)\n\nPreviously, when `interpret()` called into the compiler, it passed in a Chunk to\nbe written to. Now that the compiler creates the function object itself, we\nreturn that function. We grab it from the current compiler here:\n\n^code end-function (1 before, 1 after)\n\nAnd then return it to `compile()` like so:\n\n^code return-function (1 before, 1 after)\n\nNow is a good time to make another tweak in this function. Earlier, we added\nsome diagnostic code to have the VM dump the disassembled bytecode so we could\ndebug the compiler. We should fix that to keep working now that the generated\nchunk is wrapped in a function.\n\n^code disassemble-end (2 before, 2 after)\n\nNotice the check in here to see if the function's name is `NULL`? User-defined\nfunctions have names, but the implicit function we create for the top-level code\ndoes not, and we need to handle that gracefully even in our own diagnostic code.\nSpeaking of which:\n\n^code print-script (1 before, 1 after)\n\nThere's no way for a *user* to get a reference to the top-level function and try\nto print it, but our `DEBUG_TRACE_EXECUTION` diagnostic code that prints the entire stack can and does.\n\n\n\nBumping up a level to `compile()`, we adjust its signature.\n\n^code compile-h (2 before, 2 after)\n\nInstead of taking a chunk, now it returns a function. Over in the\nimplementation:\n\n^code compile-signature (1 after)\n\nFinally we get to some actual code. We change the very end of the function to\nthis:\n\n^code call-end-compiler (4 before, 1 after)\n\nWe get the function object from the compiler. If there were no compile errors,\nwe return it. Otherwise, we signal an error by returning `NULL`. This way, the\nVM doesn't try to execute a function that may contain invalid bytecode.\n\nEventually, we will update `interpret()` to handle the new declaration of\n`compile()`, but first we have some other changes to make.\n\n## Call Frames\n\nIt's time for a big conceptual leap. Before we can implement function\ndeclarations and calls, we need to get the VM ready to handle them. There are\ntwo main problems we need to worry about:\n\n### Allocating local variables\n\nThe compiler allocates stack slots for local variables. How should that work\nwhen the set of local variables in a program is distributed across multiple\nfunctions?\n\nOne option would be to keep them totally separate. Each function would get its\nown dedicated set of slots in the VM stack that it would own forever, even when the function isn't being called. Each\nlocal variable in the entire program would have a bit of memory in the VM that\nit keeps to itself.\n\n\n\nBelieve it or not, early programming language implementations worked this way.\nThe first Fortran compilers statically allocated memory for each variable. The\nobvious problem is that it's really inefficient. Most functions are not in the\nmiddle of being called at any point in time, so sitting on unused memory for\nthem is wasteful.\n\nThe more fundamental problem, though, is recursion. With recursion, you can be\n\"in\" multiple calls to the same function at the same time. Each needs its own memory for its local variables. In jlox, we solved\nthis by dynamically allocating memory for an environment each time a function\nwas called or a block entered. In clox, we don't want that kind of performance\ncost on every function call.\n\n\n\nInstead, our solution lies somewhere between Fortran's static allocation and\njlox's dynamic approach. The value stack in the VM works on the observation that\nlocal variables and temporaries behave in a last-in first-out fashion.\nFortunately for us, that's still true even when you add function calls into the\nmix. Here's an example:\n\n```lox\nfun first() {\n var a = 1;\n second();\n var b = 2;\n}\n\nfun second() {\n var c = 3;\n var d = 4;\n}\n\nfirst();\n```\n\nStep through the program and look at which variables are in memory at each point\nin time:\n\n\"Tracing\n\nAs execution flows through the two calls, every local variable obeys the\nprinciple that any variable declared after it will be discarded before the first\nvariable needs to be. This is true even across calls. We know we'll be done with\n`c` and `d` before we are done with `a`. It seems we should be able to allocate\nlocal variables on the VM's value stack.\n\nIdeally, we still determine *where* on the stack each variable will go at\ncompile time. That keeps the bytecode instructions for working with variables\nsimple and fast. In the above example, we could imagine doing so in a straightforward way, but that\ndoesn't always work out. Consider:\n\n\n\n```lox\nfun first() {\n var a = 1;\n second();\n var b = 2;\n second();\n}\n\nfun second() {\n var c = 3;\n var d = 4;\n}\n\nfirst();\n```\n\nIn the first call to `second()`, `c` and `d` would go into slots 1 and 2. But in\nthe second call, we need to have made room for `b`, so `c` and `d` need to be in\nslots 2 and 3. Thus the compiler can't pin down an exact slot for each local\nvariable across function calls. But *within* a given function, the *relative*\nlocations of each local variable are fixed. Variable `d` is always in the slot\nright after `c`. This is the key insight.\n\nWhen a function is called, we don't know where the top of the stack will be\nbecause it can be called from different contexts. But, wherever that top happens\nto be, we do know where all of the function's local variables will be relative\nto that starting point. So, like many problems, we solve our allocation problem\nwith a level of indirection.\n\nAt the beginning of each function call, the VM records the location of the first\nslot where that function's own locals begin. The instructions for working with\nlocal variables access them by a slot index relative to that, instead of\nrelative to the bottom of the stack like they do today. At compile time, we\ncalculate those relative slots. At runtime, we convert that relative slot to an\nabsolute stack index by adding the function call's starting slot.\n\nIt's as if the function gets a \"window\" or \"frame\" within the larger stack where\nit can store its locals. The position of the **call frame** is determined at\nruntime, but within and relative to that region, we know where to find things.\n\n\"The\n\nThe historical name for this recorded location where the function's locals start\nis a **frame pointer** because it points to the beginning of the function's call\nframe. Sometimes you hear **base pointer**, because it points to the base stack\nslot on top of which all of the function's variables live.\n\nThat's the first piece of data we need to track. Every time we call a function,\nthe VM determines the first stack slot where that function's variables begin.\n\n### Return addresses\n\nRight now, the VM works its way through the instruction stream by incrementing\nthe `ip` field. The only interesting behavior is around control flow\ninstructions which offset the `ip` by larger amounts. *Calling* a function is\npretty straightforward -- simply set `ip` to point to the first instruction in\nthat function's chunk. But what about when the function is done?\n\nThe VM needs to return back to the chunk where the\nfunction was called from and resume execution at the instruction immediately\nafter the call. Thus, for each function call, we need to track where we jump\nback to when the call completes. This is called a **return address** because\nit's the address of the instruction that the VM returns to after the call.\n\nAgain, thanks to recursion, there may be multiple return addresses for a single\nfunction, so this is a property of each *invocation* and not the function\nitself.\n\n\n\n### The call stack\n\nSo for each live function invocation -- each call that hasn't returned yet -- we\nneed to track where on the stack that function's locals begin, and where the\ncaller should resume. We'll put this, along with some other stuff, in a new\nstruct.\n\n^code call-frame (1 before, 2 after)\n\nA CallFrame represents a single ongoing function call. The `slots` field points\ninto the VM's value stack at the first slot that this function can use. I gave\nit a plural name because -- thanks to C's weird \"pointers are sort of arrays\"\nthing -- we'll treat it like an array.\n\nThe implementation of return addresses is a little different from what I\ndescribed above. Instead of storing the return address in the callee's frame,\nthe caller stores its own `ip`. When we return from a function, the VM will jump\nto the `ip` of the caller's CallFrame and resume from there.\n\nI also stuffed a pointer to the function being called in here. We'll use that to\nlook up constants and for a few other things.\n\nEach time a function is called, we create one of these structs. We could dynamically allocate them on the heap, but that's slow.\nFunction calls are a core operation, so they need to be as fast as possible.\nFortunately, we can make the same observation we made for variables: function\ncalls have stack semantics. If `first()` calls `second()`, the call to\n`second()` will complete before `first()` does.\n\n\n\nSo over in the VM, we create an array of these CallFrame structs up front and\ntreat it as a stack, like we do with the value array.\n\n^code frame-array (1 before, 1 after)\n\nThis array replaces the `chunk` and `ip` fields we used to have directly in the\nVM. Now each CallFrame has its own `ip` and its own pointer to the ObjFunction\nthat it's executing. From there, we can get to the function's chunk.\n\nThe new `frameCount` field in the VM stores the current height of the CallFrame\nstack -- the number of ongoing function calls. To keep clox simple, the array's\ncapacity is fixed. This means, as in many language implementations, there is a\nmaximum call depth we can handle. For clox, it's defined here:\n\n^code frame-max (2 before, 2 after)\n\nWe also redefine the value stack's size in terms of\nthat to make sure we have plenty of stack slots even in very deep call trees.\nWhen the VM starts up, the CallFrame stack is empty.\n\n\n\n^code reset-frame-count (1 before, 1 after)\n\nThe \"vm.h\" header needs access to ObjFunction, so we add an include.\n\n^code vm-include-object (2 before, 1 after)\n\nNow we're ready to move over to the VM's implementation file. We've got some\ngrunt work ahead of us. We've moved `ip` out of the VM struct and into\nCallFrame. We need to fix every line of code in the VM that touches `ip` to\nhandle that. Also, the instructions that access local variables by stack slot\nneed to be updated to do so relative to the current CallFrame's `slots` field.\n\nWe'll start at the top and plow through it.\n\n^code run (1 before, 1 after)\n\nFirst, we store the current topmost CallFrame in a local variable inside the main bytecode execution function.\nThen we replace the bytecode access macros with versions that access `ip`\nthrough that variable.\n\n\n\nNow onto each instruction that needs a little tender loving care.\n\n^code push-local (2 before, 1 after)\n\nPreviously, `OP_GET_LOCAL` read the given local slot directly from the VM's\nstack array, which meant it indexed the slot starting from the bottom of the\nstack. Now, it accesses the current frame's `slots` array, which means it\naccesses the given numbered slot relative to the beginning of that frame.\n\nSetting a local variable works the same way.\n\n^code set-local (2 before, 1 after)\n\nThe jump instructions used to modify the VM's `ip` field. Now, they do the same\nfor the current frame's `ip`.\n\n^code jump (2 before, 1 after)\n\nSame with the conditional jump:\n\n^code jump-if-false (2 before, 1 after)\n\nAnd our backward-jumping loop instruction:\n\n^code loop (2 before, 1 after)\n\nWe have some diagnostic code that prints each instruction as it executes to help\nus debug our VM. That needs to work with the new structure too.\n\n^code trace-execution (1 before, 1 after)\n\nInstead of passing in the VM's `chunk` and `ip` fields, now we read from the\ncurrent CallFrame.\n\nYou know, that wasn't too bad, actually. Most instructions just use the macros\nso didn't need to be touched. Next, we jump up a level to the code that calls\n`run()`.\n\n^code interpret-stub (1 before, 2 after)\n\nWe finally get to wire up our earlier compiler changes to the back-end changes\nwe just made. First, we pass the source code to the compiler. It returns us a\nnew ObjFunction containing the compiled top-level code. If we get `NULL` back,\nit means there was some compile-time error which the compiler has already\nreported. In that case, we bail out since we can't run anything.\n\nOtherwise, we store the function on the stack and prepare an initial CallFrame\nto execute its code. Now you can see why the compiler sets aside stack slot zero\n-- that stores the function being called. In the new CallFrame, we point to the\nfunction, initialize its `ip` to point to the beginning of the function's\nbytecode, and set up its stack window to start at the very bottom of the VM's\nvalue stack.\n\nThis gets the interpreter ready to start executing code. After finishing, the VM\nused to free the hardcoded chunk. Now that the ObjFunction owns that code, we\ndon't need to do that anymore, so the end of `interpret()` is simply this:\n\n^code end-interpret (2 before, 1 after)\n\nThe last piece of code referring to the old VM fields is `runtimeError()`. We'll\nrevisit that later in the chapter, but for now let's change it to this:\n\n^code runtime-error-temp (2 before, 1 after)\n\nInstead of reading the chunk and `ip` directly from the VM, it pulls those from\nthe topmost CallFrame on the stack. That should get the function working again\nand behaving as it did before.\n\nAssuming we did all of that correctly, we got clox back to a runnable\nstate. Fire it up and it does... exactly what it did before. We haven't added\nany new features yet, so this is kind of a let down. But all of the\ninfrastructure is there and ready for us now. Let's take advantage of it.\n\n## Function Declarations\n\nBefore we can do call expressions, we need something to call, so we'll do\nfunction declarations first. The fun starts with a\nkeyword.\n\n\n\n^code match-fun (1 before, 1 after)\n\nThat passes control to here:\n\n^code fun-declaration\n\nFunctions are first-class values, and a function declaration simply creates and\nstores one in a newly declared variable. So we parse the name just like any\nother variable declaration. A function declaration at the top level will bind\nthe function to a global variable. Inside a block or other function, a function\ndeclaration creates a local variable.\n\nIn an earlier chapter, I explained how variables [get defined in two\nstages][stage]. This ensures you can't access a variable's value inside the\nvariable's own initializer. That would be bad because the variable doesn't\n*have* a value yet.\n\n[stage]: local-variables.html#another-scope-edge-case\n\nFunctions don't suffer from this problem. It's safe for a function to refer to\nits own name inside its body. You can't *call* the function and execute the body\nuntil after it's fully defined, so you'll never see the variable in an\nuninitialized state. Practically speaking, it's useful to allow this in order to\nsupport recursive local functions.\n\nTo make that work, we mark the function declaration's variable \"initialized\" as\nsoon as we compile the name, before we compile the body. That way the name can\nbe referenced inside the body without generating an error.\n\nWe do need one check, though.\n\n^code check-depth (1 before, 1 after)\n\nBefore, we called `markInitialized()` only when we already knew we were in a\nlocal scope. Now, a top-level function declaration will also call this function.\nWhen that happens, there is no local variable to mark initialized -- the\nfunction is bound to a global variable.\n\nNext, we compile the function itself -- its parameter list and block body. For\nthat, we use a separate helper function. That helper generates code that\nleaves the resulting function object on top of the stack. After that, we call\n`defineVariable()` to store that function back into the variable we declared for\nit.\n\nI split out the code to compile the parameters and body because we'll reuse it\nlater for parsing method declarations inside classes. Let's build it\nincrementally, starting with this:\n\n^code compile-function\n\n\n\nFor now, we won't worry about parameters. We parse an empty pair of parentheses\nfollowed by the body. The body starts with a left curly brace, which we parse\nhere. Then we call our existing `block()` function, which knows how to compile\nthe rest of a block including the closing brace.\n\n### A stack of compilers\n\nThe interesting parts are the compiler stuff at the top and bottom. The Compiler\nstruct stores data like which slots are owned by which local variables, how many\nblocks of nesting we're currently in, etc. All of that is specific to a single\nfunction. But now the front end needs to handle compiling multiple functions\nnested within each other.\n\n\n\nThe trick for managing that is to create a separate Compiler for each function\nbeing compiled. When we start compiling a function declaration, we create a new\nCompiler on the C stack and initialize it. `initCompiler()` sets that Compiler\nto be the current one. Then, as we compile the body, all of the functions that\nemit bytecode write to the chunk owned by the new Compiler's function.\n\nAfter we reach the end of the function's block body, we call `endCompiler()`.\nThat yields the newly compiled function object, which we store as a constant in\nthe *surrounding* function's constant table. But, wait, how do we get back to\nthe surrounding function? We lost it when `initCompiler()` overwrote the current\ncompiler pointer.\n\nWe fix that by treating the series of nested Compiler structs as a stack. Unlike\nthe Value and CallFrame stacks in the VM, we won't use an array. Instead, we use\na linked list. Each Compiler points back to the Compiler for the function that\nencloses it, all the way back to the root Compiler for the top-level code.\n\n^code enclosing-field (2 before, 1 after)\n\nInside the Compiler struct, we can't reference the Compiler *typedef* since that\ndeclaration hasn't finished yet. Instead, we give a name to the struct itself\nand use that for the field's type. C is weird.\n\nWhen initializing a new Compiler, we capture the about-to-no-longer-be-current\none in that pointer.\n\n^code store-enclosing (1 before, 1 after)\n\nThen when a Compiler finishes, it pops itself off the stack by restoring the\nprevious compiler to be the new current one.\n\n^code restore-enclosing (2 before, 1 after)\n\nNote that we don't even need to dynamically\nallocate the Compiler structs. Each is stored as a local variable in the C stack\n-- either in `compile()` or `function()`. The linked list of Compilers threads\nthrough the C stack. The reason we can get an unbounded number of them is\nbecause our compiler uses recursive descent, so `function()` ends up calling\nitself recursively when you have nested function declarations.\n\n\n\n### Function parameters\n\nFunctions aren't very useful if you can't pass arguments to them, so let's do\nparameters next.\n\n^code parameters (1 before, 1 after)\n\nSemantically, a parameter is simply a local variable declared in the outermost\nlexical scope of the function body. We get to use the existing compiler support\nfor declaring named local variables to parse and compile parameters. Unlike\nlocal variables, which have initializers, there's no code here to initialize the\nparameter's value. We'll see how they are initialized later when we do argument\npassing in function calls.\n\nWhile we're at it, we note the function's arity by counting how many parameters\nwe parse. The other piece of metadata we store with a function is its name. When\ncompiling a function declaration, we call `initCompiler()` right after we parse\nthe function's name. That means we can grab the name right then from the\nprevious token.\n\n^code init-function-name (1 before, 2 after)\n\nNote that we're careful to create a copy of the name string. Remember, the\nlexeme points directly into the original source code string. That string may get\nfreed once the code is finished compiling. The function object we create in the\ncompiler outlives the compiler and persists until runtime. So it needs its own\nheap-allocated name string that it can keep around.\n\nRad. Now we can compile function declarations, like this:\n\n```lox\nfun areWeHavingItYet() {\n print \"Yes we are!\";\n}\n\nprint areWeHavingItYet;\n```\n\nWe just can't do anything useful with them.\n\n\n\n## Function Calls\n\nBy the end of this section, we'll start to see some interesting behavior. The\nnext step is calling functions. We don't usually think of it this way, but a\nfunction call expression is kind of an infix `(` operator. You have a\nhigh-precedence expression on the left for the thing being called -- usually\njust a single identifier. Then the `(` in the middle, followed by the argument\nexpressions separated by commas, and a final `)` to wrap it up at the end.\n\nThat odd grammatical perspective explains how to hook the syntax into our\nparsing table.\n\n^code infix-left-paren (1 before, 1 after)\n\nWhen the parser encounters a left parenthesis following an expression, it\ndispatches to a new parser function.\n\n^code compile-call\n\nWe've already consumed the `(` token, so next we compile the arguments using a\nseparate `argumentList()` helper. That function returns the number of arguments\nit compiled. Each argument expression generates code that leaves its value on\nthe stack in preparation for the call. After that, we emit a new `OP_CALL`\ninstruction to invoke the function, using the argument count as an operand.\n\nWe compile the arguments using this friend:\n\n^code argument-list\n\nThat code should look familiar from jlox. We chew through arguments as long as\nwe find commas after each expression. Once we run out, we consume the final\nclosing parenthesis and we're done.\n\nWell, almost. Back in jlox, we added a compile-time check that you don't pass\nmore than 255 arguments to a call. At the time, I said that was because clox\nwould need a similar limit. Now you can see why -- since we stuff the argument\ncount into the bytecode as a single-byte operand, we can only go up to 255. We\nneed to verify that in this compiler too.\n\n^code arg-limit (1 before, 1 after)\n\nThat's the front end. Let's skip over to the back end, with a quick stop in the\nmiddle to declare the new instruction.\n\n^code op-call (1 before, 1 after)\n\n### Binding arguments to parameters\n\nBefore we get to the implementation, we should think about what the stack looks\nlike at the point of a call and what we need to do from there. When we reach the\ncall instruction, we have already executed the expression for the function being\ncalled, followed by its arguments. Say our program looks like this:\n\n```lox\nfun sum(a, b, c) {\n return a + b + c;\n}\n\nprint 4 + sum(5, 6, 7);\n```\n\nIf we pause the VM right on the `OP_CALL` instruction for that call to `sum()`,\nthe stack looks like this:\n\n\"Stack:\n\nPicture this from the perspective of `sum()` itself. When the compiler compiled\n`sum()`, it automatically allocated slot zero. Then, after that, it allocated\nlocal slots for the parameters `a`, `b`, and `c`, in order. To perform a call to\n`sum()`, we need a CallFrame initialized with the function being called and a\nregion of stack slots that it can use. Then we need to collect the arguments\npassed to the function and get them into the corresponding slots for the\nparameters.\n\nWhen the VM starts executing the body of `sum()`, we want its stack window to\nlook like this:\n\n\"The\n\nDo you notice how the argument slots that the caller sets up and the parameter\nslots the callee needs are both in exactly the right order? How convenient! This\nis no coincidence. When I talked about each CallFrame having its own window into\nthe stack, I never said those windows must be *disjoint*. There's nothing\npreventing us from overlapping them, like this:\n\n\"The\n\nThe top of the caller's stack contains the function\nbeing called followed by the arguments in order. We know the caller doesn't have\nany other slots above those in use because any temporaries needed when\nevaluating argument expressions have been discarded by now. The bottom of the\ncallee's stack overlaps so that the parameter slots exactly line up with where\nthe argument values already live.\n\n\n\nThis means that we don't need to do *any* work to \"bind an argument to a\nparameter\". There's no copying values between slots or across environments. The\narguments are already exactly where they need to be. It's hard to beat that for\nperformance.\n\nTime to implement the call instruction.\n\n^code interpret-call (1 before, 1 after)\n\nWe need to know the function being called and the number of arguments passed to\nit. We get the latter from the instruction's operand. That also tells us where\nto find the function on the stack by counting past the argument slots from the\ntop of the stack. We hand that data off to a separate `callValue()` function. If\nthat returns `false`, it means the call caused some sort of runtime error. When\nthat happens, we abort the interpreter.\n\nIf `callValue()` is successful, there will be a new frame on the CallFrame stack\nfor the called function. The `run()` function has its own cached pointer to the\ncurrent frame, so we need to update that.\n\n^code update-frame-after-call (2 before, 1 after)\n\nSince the bytecode dispatch loop reads from that `frame` variable, when the VM\ngoes to execute the next instruction, it will read the `ip` from the newly\ncalled function's CallFrame and jump to its code. The work for executing that\ncall begins here:\n\n^code call-value\n\n\n\nThere's more going on here than just initializing a new CallFrame. Because Lox\nis dynamically typed, there's nothing to prevent a user from writing bad code\nlike:\n\n```lox\nvar notAFunction = 123;\nnotAFunction();\n```\n\nIf that happens, the runtime needs to safely report an error and halt. So the\nfirst thing we do is check the type of the value that we're trying to call. If\nit's not a function, we error out. Otherwise, the actual call happens here:\n\n^code call\n\nThis simply initializes the next CallFrame on the stack. It stores a pointer to\nthe function being called and points the frame's `ip` to the beginning of the\nfunction's bytecode. Finally, it sets up the `slots` pointer to give the frame\nits window into the stack. The arithmetic there ensures that the arguments\nalready on the stack line up with the function's parameters:\n\n\"The\n\nThe funny little `- 1` is to account for stack slot zero which the compiler set\naside for when we add methods later. The parameters start at slot one so we\nmake the window start one slot earlier to align them with the arguments.\n\nBefore we move on, let's add the new instruction to our disassembler.\n\n^code disassemble-call (1 before, 1 after)\n\nAnd one more quick side trip. Now that we have a handy function for initiating a\nCallFrame, we may as well use it to set up the first frame for executing the\ntop-level code.\n\n^code interpret (1 before, 2 after)\n\nOK, now back to calls...\n\n### Runtime error checking\n\nThe overlapping stack windows work based on the assumption that a call passes\nexactly one argument for each of the function's parameters. But, again, because\nLox ain't statically typed, a foolish user could pass too many or too few\narguments. In Lox, we've defined that to be a runtime error, which we report\nlike so:\n\n^code check-arity (1 before, 1 after)\n\nPretty straightforward. This is why we store the arity of each function inside\nthe ObjFunction for it.\n\nThere's another error we need to report that's less to do with the user's\nfoolishness than our own. Because the CallFrame array has a fixed size, we need\nto ensure a deep call chain doesn't overflow it.\n\n^code check-overflow (2 before, 1 after)\n\nIn practice, if a program gets anywhere close to this limit, there's most likely\na bug in some runaway recursive code.\n\n### Printing stack traces\n\nWhile we're on the subject of runtime errors, let's spend a little time making\nthem more useful. Stopping on a runtime error is important to prevent the VM\nfrom crashing and burning in some ill-defined way. But simply aborting doesn't\nhelp the user fix their code that *caused* that error.\n\nThe classic tool to aid debugging runtime failures is a **stack trace** -- a\nprint out of each function that was still executing when the program died, and\nwhere the execution was at the point that it died. Now that we have a call stack\nand we've conveniently stored each function's name, we can show that entire\nstack when a runtime error disrupts the harmony of the user's existence. It\nlooks like this:\n\n^code runtime-error-stack (2 before, 2 after)\n\n\n\nAfter printing the error message itself, we walk the call stack from top (the most recently called function) to bottom (the\ntop-level code). For each frame, we find the line number that corresponds to the\ncurrent `ip` inside that frame's function. Then we print that line number along\nwith the function name.\n\n\n\nFor example, if you run this broken program:\n\n```lox\nfun a() { b(); }\nfun b() { c(); }\nfun c() {\n c(\"too\", \"many\");\n}\n\na();\n```\n\nIt prints out:\n\n```text\nExpected 0 arguments but got 2.\n[line 4] in c()\n[line 2] in b()\n[line 1] in a()\n[line 7] in script\n```\n\nThat doesn't look too bad, does it?\n\n### Returning from functions\n\nWe're getting close. We can call functions, and the VM will execute them. But we\ncan't *return* from them yet. We've had an `OP_RETURN` instruction for quite\nsome time, but it's always had some kind of temporary code hanging out in it\njust to get us out of the bytecode loop. The time has arrived for a real\nimplementation.\n\n^code interpret-return (1 before, 1 after)\n\nWhen a function returns a value, that value will be on top of the stack. We're\nabout to discard the called function's entire stack window, so we pop that\nreturn value off and hang on to it. Then we discard the CallFrame for the\nreturning function. If that was the very last CallFrame, it means we've finished\nexecuting the top-level code. The entire program is done, so we pop the main\nscript function from the stack and then exit the interpreter.\n\nOtherwise, we discard all of the slots the callee was using for its parameters\nand local variables. That includes the same slots the caller used to pass the\narguments. Now that the call is done, the caller doesn't need them anymore. This\nmeans the top of the stack ends up right at the beginning of the returning\nfunction's stack window.\n\nWe push the return value back onto the stack at that new, lower location. Then\nwe update the `run()` function's cached pointer to the current frame. Just like\nwhen we began a call, on the next iteration of the bytecode dispatch loop, the\nVM will read `ip` from that frame, and execution will jump back to the caller,\nright where it left off, immediately after the `OP_CALL` instruction.\n\n\"Each\n\nNote that we assume here that the function *did* actually return a value, but\na function can implicitly return by reaching the end of its body:\n\n```lox\nfun noReturn() {\n print \"Do stuff\";\n // No return here.\n}\n\nprint noReturn(); // ???\n```\n\nWe need to handle that correctly too. The language is specified to implicitly\nreturn `nil` in that case. To make that happen, we add this:\n\n^code return-nil (1 before, 2 after)\n\nThe compiler calls `emitReturn()` to write the `OP_RETURN` instruction at the\nend of a function body. Now, before that, it emits an instruction to push `nil`\nonto the stack. And with that, we have working function calls! They can even\ntake parameters! It almost looks like we know what we're doing here.\n\n## Return Statements\n\nIf you want a function that returns something other than the implicit `nil`, you\nneed a `return` statement. Let's get that working.\n\n^code match-return (1 before, 1 after)\n\nWhen the compiler sees a `return` keyword, it goes here:\n\n^code return-statement\n\nThe return value expression is optional, so the parser looks for a semicolon\ntoken to tell if a value was provided. If there is no return value, the\nstatement implicitly returns `nil`. We implement that by calling `emitReturn()`,\nwhich emits an `OP_NIL` instruction. Otherwise, we compile the return value\nexpression and return it with an `OP_RETURN` instruction.\n\nThis is the same `OP_RETURN` instruction we've already implemented -- we don't\nneed any new runtime code. This is quite a difference from jlox. There, we had\nto use exceptions to unwind the stack when a `return` statement was executed.\nThat was because you could return from deep inside some nested blocks. Since\njlox recursively walks the AST, that meant there were a bunch of Java method\ncalls we needed to escape out of.\n\nOur bytecode compiler flattens that all out. We do recursive descent during\nparsing, but at runtime, the VM's bytecode dispatch loop is completely flat.\nThere is no recursion going on at the C level at all. So returning, even from\nwithin some nested blocks, is as straightforward as returning from the end of\nthe function's body.\n\nWe're not totally done, though. The new `return` statement gives us a new\ncompile error to worry about. Returns are useful for returning from functions\nbut the top level of a Lox program is imperative code too. You shouldn't be able\nto return from there.\n\n```lox\nreturn \"What?!\";\n```\n\n\n\nWe've specified that it's a compile error to have a `return` statement outside\nof any function, which we implement like so:\n\n^code return-from-script (1 before, 1 after)\n\nThis is one of the reasons we added that FunctionType enum to the compiler.\n\n## Native Functions\n\nOur VM is getting more powerful. We've got functions, calls, parameters,\nreturns. You can define lots of different functions that can call each other in\ninteresting ways. But, ultimately, they can't really *do* anything. The only\nuser-visible thing a Lox program can do, regardless of its complexity, is print.\nTo add more capabilities, we need to expose them to the user.\n\nA programming language implementation reaches out and touches the material world\nthrough **native functions**. If you want to be able to write programs that\ncheck the time, read user input, or access the file system, we need to add\nnative functions -- callable from Lox but implemented in C -- that expose those\ncapabilities.\n\nAt the language level, Lox is fairly complete -- it's got closures, classes,\ninheritance, and other fun stuff. One reason it feels like a toy language is\nbecause it has almost no native capabilities. We could turn it into a real\nlanguage by adding a long list of them.\n\nHowever, grinding through a pile of OS operations isn't actually very\neducational. Once you've seen how to bind one piece of C code to Lox, you get\nthe idea. But you do need to see *one*, and even a single native function\nrequires us to build out all the machinery for interfacing Lox with C. So we'll\ngo through that and do all the hard work. Then, when that's done, we'll add one\ntiny native function just to prove that it works.\n\nThe reason we need new machinery is because, from the implementation's\nperspective, native functions are different from Lox functions. When they are\ncalled, they don't push a CallFrame, because there's no bytecode code for that\nframe to point to. They have no bytecode chunk. Instead, they somehow reference\na piece of native C code.\n\nWe handle this in clox by defining native functions as an entirely different\nobject type.\n\n^code obj-native (1 before, 2 after)\n\nThe representation is simpler than ObjFunction -- merely an Obj header and a\npointer to the C function that implements the native behavior. The native\nfunction takes the argument count and a pointer to the first argument on the\nstack. It accesses the arguments through that pointer. Once it's done, it\nreturns the result value.\n\nAs always, a new object type carries some accoutrements with it. To create an\nObjNative, we declare a constructor-like function.\n\n^code new-native-h (1 before, 1 after)\n\nWe implement that like so:\n\n^code new-native\n\nThe constructor takes a C function pointer to wrap in an ObjNative. It sets up\nthe object header and stores the function. For the header, we need a new object\ntype.\n\n^code obj-type-native (2 before, 2 after)\n\nThe VM also needs to know how to deallocate a native function object.\n\n^code free-native (1 before, 1 after)\n\nThere isn't much here since ObjNative doesn't own any extra memory. The other\ncapability all Lox objects support is being printed.\n\n^code print-native (1 before, 1 after)\n\nIn order to support dynamic typing, we have a macro to see if a value is a\nnative function.\n\n^code is-native (1 before, 1 after)\n\nAssuming that returns true, this macro extracts the C function pointer from a\nValue representing a native function:\n\n^code as-native (1 before, 1 after)\n\nAll of this baggage lets the VM treat native functions like any other object.\nYou can store them in variables, pass them around, throw them birthday parties,\netc. Of course, the operation we actually care about is *calling* them -- using\none as the left-hand operand in a call expression.\n\nOver in `callValue()` we add another type case.\n\n^code call-native (2 before, 1 after)\n\nIf the object being called is a native function, we invoke the C function right\nthen and there. There's no need to muck with CallFrames or anything. We just\nhand off to C, get the result, and stuff it back in the stack. This makes native\nfunctions as fast as we can get.\n\nWith this, users should be able to call native functions, but there aren't any\nto call. Without something like a foreign function interface, users can't define\ntheir own native functions. That's our job as VM implementers. We'll start with\na helper to define a new native function exposed to Lox programs.\n\n^code define-native\n\nIt takes a pointer to a C function and the name it will be known as in Lox.\nWe wrap the function in an ObjNative and then store that in a global variable\nwith the given name.\n\nYou're probably wondering why we push and pop the name and function on the\nstack. That looks weird, right? This is the kind of stuff you have to worry\nabout when garbage collection gets involved. Both\n`copyString()` and `newNative()` dynamically allocate memory. That means once we\nhave a GC, they can potentially trigger a collection. If that happens, we need\nto ensure the collector knows we're not done with the name and ObjFunction so\nthat it doesn't free them out from under us. Storing them on the value stack\naccomplishes that.\n\n\n\nIt feels silly, but after all of that work, we're going to add only one\nlittle native function.\n\n^code clock-native\n\nThis returns the elapsed time since the program started running, in seconds. It's\nhandy for benchmarking Lox programs. In Lox, we'll name it `clock()`.\n\n^code define-native-clock (1 before, 1 after)\n\nTo get to the C standard library `clock()` function, the \"vm\" module needs an\ninclude.\n\n^code vm-include-time (1 before, 2 after)\n\nThat was a lot of material to work through, but we did it! Type this in and try\nit out:\n\n```lox\nfun fib(n) {\n if (n < 2) return n;\n return fib(n - 2) + fib(n - 1);\n}\n\nvar start = clock();\nprint fib(35);\nprint clock() - start;\n```\n\nWe can write a really inefficient recursive Fibonacci function. Even better, we\ncan measure just *how* inefficient it is. This is, of\ncourse, not the smartest way to calculate a Fibonacci number. But it is a good\nway to stress test a language implementation's support for function calls. On my\nmachine, running this in clox is about five times faster than in jlox. That's\nquite an improvement.\n\n\n\n
\n\n## Challenges\n\n1. Reading and writing the `ip` field is one of the most frequent operations\n inside the bytecode loop. Right now, we access it through a pointer to the\n current CallFrame. That requires a pointer indirection which may force the\n CPU to bypass the cache and hit main memory. That can be a real performance\n sink.\n\n Ideally, we'd keep the `ip` in a native CPU register. C doesn't let us\n *require* that without dropping into inline assembly, but we can structure\n the code to encourage the compiler to make that optimization. If we store\n the `ip` directly in a C local variable and mark it `register`, there's a\n good chance the C compiler will accede to our polite request.\n\n This does mean we need to be careful to load and store the local `ip` back\n into the correct CallFrame when starting and ending function calls.\n Implement this optimization. Write a couple of benchmarks and see how it\n affects the performance. Do you think the extra code complexity is worth it?\n\n2. Native function calls are fast in part because we don't validate that the\n call passes as many arguments as the function expects. We really should, or\n an incorrect call to a native function without enough arguments could cause\n the function to read uninitialized memory. Add arity checking.\n\n3. Right now, there's no way for a native function to signal a runtime error.\n In a real implementation, this is something we'd need to support because\n native functions live in the statically typed world of C but are called\n from dynamically typed Lox land. If a user, say, tries to pass a string to\n `sqrt()`, that native function needs to report a runtime error.\n\n Extend the native function system to support that. How does this capability\n affect the performance of native calls?\n\n4. Add some more native functions to do things you find useful. Write some\n programs using those. What did you add? How do they affect the feel of the\n language and how practical it is?\n\n
\n" }, { "path": "book/chunks-of-bytecode.md", "content": "> If you find that you're spending almost all your time on theory, start turning\n> some attention to practical things; it will improve your theories. If you find\n> that you're spending almost all your time on practice, start turning some\n> attention to theoretical things; it will improve your practice.\n>\n> Donald Knuth\n\nWe already have ourselves a complete implementation of Lox with jlox, so why\nisn't the book over yet? Part of this is because jlox relies on the JVM to do lots of things for us. If we want to understand\nhow an interpreter works all the way down to the metal, we need to build those\nbits and pieces ourselves.\n\n\n\nAn even more fundamental reason that jlox isn't sufficient is that it's too damn\nslow. A tree-walk interpreter is fine for some kinds of high-level, declarative\nlanguages. But for a general-purpose, imperative language -- even a \"scripting\"\nlanguage like Lox -- it won't fly. Take this little script:\n\n```lox\nfun fib(n) {\n if (n < 2) return n;\n return fib(n - 1) + fib(n - 2); // [fib]\n}\n\nvar before = clock();\nprint fib(40);\nvar after = clock();\nprint after - before;\n```\n\n\n\nOn my laptop, that takes jlox about 72 seconds to execute. An equivalent C\nprogram finishes in half a second. Our dynamically typed scripting language is\nnever going to be as fast as a statically typed language with manual memory\nmanagement, but we don't need to settle for more than *two orders of magnitude*\nslower.\n\nWe could take jlox and run it in a profiler and start tuning and tweaking\nhotspots, but that will only get us so far. The execution model -- walking the\nAST -- is fundamentally the wrong design. We can't micro-optimize that to the\nperformance we want any more than you can polish an AMC Gremlin into an SR-71\nBlackbird.\n\nWe need to rethink the core model. This chapter introduces that model, bytecode,\nand begins our new interpreter, clox.\n\n## Bytecode?\n\nIn engineering, few choices are without trade-offs. To best understand why we're\ngoing with bytecode, let's stack it up against a couple of alternatives.\n\n### Why not walk the AST?\n\nOur existing interpreter has a couple of things going for it:\n\n* Well, first, we already wrote it. It's done. And the main reason it's done\n is because this style of interpreter is *really simple to implement*. The\n runtime representation of the code directly maps to the syntax. It's\n virtually effortless to get from the parser to the data structures we need\n at runtime.\n\n* It's *portable*. Our current interpreter is written in Java and runs on any\n platform Java supports. We could write a new implementation in C using the\n same approach and compile and run our language on basically every platform\n under the sun.\n\nThose are real advantages. But, on the other hand, it's *not memory-efficient*.\nEach piece of syntax becomes an AST node. A tiny Lox expression like `1 + 2`\nturns into a slew of objects with lots of pointers between them, something like:\n\n\n\n\n\n\"The\n\nEach of those pointers adds an extra 32 or 64 bits of overhead to the object.\nWorse, sprinkling our data across the heap in a loosely connected web of objects\ndoes bad things for *spatial locality*.\n\n\n\nModern CPUs process data way faster than they can pull it from RAM. To\ncompensate for that, chips have multiple layers of caching. If a piece of memory\nit needs is already in the cache, it can be loaded more quickly. We're talking\nupwards of 100 *times* faster.\n\nHow does data get into that cache? The machine speculatively stuffs things in\nthere for you. Its heuristic is pretty simple. Whenever the CPU reads a bit of\ndata from RAM, it pulls in a whole little bundle of adjacent bytes and stuffs\nthem in the cache.\n\nIf our program next requests some data close enough to be inside that cache\nline, our CPU runs like a well-oiled conveyor belt in a factory. We *really*\nwant to take advantage of this. To use the cache effectively, the way we\nrepresent code in memory should be dense and ordered like it's read.\n\nNow look up at that tree. Those sub-objects could be *anywhere*. Every step the tree-walker takes where it\nfollows a reference to a child node may step outside the bounds of the cache and\nforce the CPU to stall until a new lump of data can be slurped in from RAM. Just\nthe *overhead* of those tree nodes with all of their pointer fields and object\nheaders tends to push objects away from each other and out of the cache.\n\n\n\nOur AST walker has other overhead too around interface dispatch and the Visitor\npattern, but the locality issues alone are enough to justify a better code\nrepresentation.\n\n### Why not compile to native code?\n\nIf you want to go *real* fast, you want to get all of those layers of\nindirection out of the way. Right down to the metal. Machine code. It even\n*sounds* fast. *Machine code.*\n\nCompiling directly to the native instruction set the chip supports is what the\nfastest languages do. Targeting native code has been the most efficient option\nsince way back in the early days when engineers actually handwrote programs in machine code.\n\n\n\nIf you've never written any machine code, or its slightly more human-palatable\ncousin assembly code before, I'll give you the gentlest of introductions. Native\ncode is a dense series of operations, encoded directly in binary. Each\ninstruction is between one and a few bytes long, and is almost mind-numbingly\nlow level. \"Move a value from this address to this register.\" \"Add the integers\nin these two registers.\" Stuff like that.\n\nThe CPU cranks through the instructions, decoding and executing each one in\norder. There is no tree structure like our AST, and control flow is handled by\njumping from one point in the code directly to another. No indirection, no\noverhead, no unnecessary skipping around or chasing pointers.\n\nLightning fast, but that performance comes at a cost. First of all, compiling to\nnative code ain't easy. Most chips in wide use today have sprawling Byzantine\narchitectures with heaps of instructions that accreted over decades. They\nrequire sophisticated register allocation, pipelining, and instruction\nscheduling.\n\nAnd, of course, you've thrown portability out. Spend a\nfew years mastering some architecture and that still only gets you onto *one* of\nthe several popular instruction sets out there. To get your language on all of\nthem, you need to learn all of their instruction sets and write a separate back\nend for each one.\n\n\n\n### What is bytecode?\n\nFix those two points in your mind. On one end, a tree-walk interpreter is\nsimple, portable, and slow. On the other, native code is complex and\nplatform-specific but fast. Bytecode sits in the middle. It retains the\nportability of a tree-walker -- we won't be getting our hands dirty with\nassembly code in this book. It sacrifices *some* simplicity to get a performance\nboost in return, though not as fast as going fully native.\n\nStructurally, bytecode resembles machine code. It's a dense, linear sequence of\nbinary instructions. That keeps overhead low and plays nice with the cache.\nHowever, it's a much simpler, higher-level instruction set than any real chip\nout there. (In many bytecode formats, each instruction is only a single byte\nlong, hence \"bytecode\".)\n\nImagine you're writing a native compiler from some source language and you're\ngiven carte blanche to define the easiest possible architecture to target.\nBytecode is kind of like that. It's an idealized fantasy instruction set that\nmakes your life as the compiler writer easier.\n\nThe problem with a fantasy architecture, of course, is that it doesn't exist. We\nsolve that by writing an *emulator* -- a simulated chip written in software that\ninterprets the bytecode one instruction at a time. A *virtual machine (VM)*, if\nyou will.\n\nThat emulation layer adds overhead, which is a key\nreason bytecode is slower than native code. But in return, it gives us\nportability. Write our VM in a language like C that is already supported on all\nthe machines we care about, and we can run our emulator on top of any hardware\nwe like.\n\n\n\nThis is the path we'll take with our new interpreter, clox. We'll follow in the\nfootsteps of the main implementations of Python, Ruby, Lua, OCaml, Erlang, and\nothers. In many ways, our VM's design will parallel the structure of our\nprevious interpreter:\n\n\"Phases\n\nOf course, we won't implement the phases strictly in order. Like our previous\ninterpreter, we'll bounce around, building up the implementation one language\nfeature at a time. In this chapter, we'll get the skeleton of the application in\nplace and create the data structures needed to store and represent a chunk of\nbytecode.\n\n## Getting Started\n\nWhere else to begin, but at `main()`? Fire up your\ntrusty text editor and start typing.\n\n\n\n^code main-c\n\nFrom this tiny seed, we will grow our entire VM. Since C provides us with so\nlittle, we first need to spend some time amending the soil. Some of that goes\ninto this header:\n\n^code common-h\n\nThere are a handful of types and constants we'll use throughout the interpreter,\nand this is a convenient place to put them. For now, it's the venerable `NULL`,\n`size_t`, the nice C99 Boolean `bool`, and explicit-sized integer types --\n`uint8_t` and friends.\n\n## Chunks of Instructions\n\nNext, we need a module to define our code representation. I've been using\n\"chunk\" to refer to sequences of bytecode, so let's make that the official name\nfor that module.\n\n^code chunk-h\n\nIn our bytecode format, each instruction has a one-byte **operation code**\n(universally shortened to **opcode**). That number controls what kind of\ninstruction we're dealing with -- add, subtract, look up variable, etc. We\ndefine those here:\n\n^code op-enum (1 before, 2 after)\n\nFor now, we start with a single instruction, `OP_RETURN`. When we have a\nfull-featured VM, this instruction will mean \"return from the current function\".\nI admit this isn't exactly useful yet, but we have to start somewhere, and this\nis a particularly simple instruction, for reasons we'll get to later.\n\n### A dynamic array of instructions\n\nBytecode is a series of instructions. Eventually, we'll store some other data\nalong with the instructions, so let's go ahead and create a struct to hold it\nall.\n\n^code chunk-struct (1 before, 2 after)\n\nAt the moment, this is simply a wrapper around an array of bytes. Since we don't\nknow how big the array needs to be before we start compiling a chunk, it must be\ndynamic. Dynamic arrays are one of my favorite data structures. That sounds like\nclaiming vanilla is my favorite ice cream flavor, but\nhear me out. Dynamic arrays provide:\n\n\n\n* Cache-friendly, dense storage\n\n* Constant-time indexed element lookup\n\n* Constant-time appending to the end of the array\n\nThose features are exactly why we used dynamic arrays all the time in jlox under\nthe guise of Java's ArrayList class. Now that we're in C, we get to roll our\nown. If you're rusty on dynamic arrays, the idea is pretty simple. In addition\nto the array itself, we keep two numbers: the number of elements in the array we\nhave allocated (\"capacity\") and how many of those allocated entries are actually\nin use (\"count\").\n\n^code count-and-capacity (1 before, 2 after)\n\nWhen we add an element, if the count is less than the capacity, then there is\nalready available space in the array. We store the new element right in there\nand bump the count.\n\n\"Storing\n\nIf we have no spare capacity, then the process is a little more involved.\n\n\"Growing\n\n1. Allocate a new array with more capacity.\n2. Copy the existing elements from the old array to the new one.\n3. Store the new `capacity`.\n4. Delete the old array.\n5. Update `code` to point to the new array.\n6. Store the element in the new array now that there is room.\n7. Update the `count`.\n\n\n\nWe have our struct ready, so let's implement the functions to work with it. C\ndoesn't have constructors, so we declare a function to initialize a new chunk.\n\n^code init-chunk-h (1 before, 2 after)\n\nAnd implement it thusly:\n\n^code chunk-c\n\nThe dynamic array starts off completely empty. We don't even allocate a raw\narray yet. To append a byte to the end of the chunk, we use a new function.\n\n^code write-chunk-h (1 before, 2 after)\n\nThis is where the interesting work happens.\n\n^code write-chunk\n\nThe first thing we need to do is see if the current array already has capacity\nfor the new byte. If it doesn't, then we first need to grow the array to make\nroom. (We also hit this case on the very first write when the array is `NULL`\nand `capacity` is 0.)\n\nTo grow the array, first we figure out the new capacity and grow the array to\nthat size. Both of those lower-level memory operations are defined in a new\nmodule.\n\n^code chunk-c-include-memory (1 before, 2 after)\n\nThis is enough to get us started.\n\n^code memory-h\n\nThis macro calculates a new capacity based on a given current capacity. In order\nto get the performance we want, the important part is that it *scales* based on\nthe old size. We grow by a factor of two, which is pretty typical. 1.5× is\nanother common choice.\n\nWe also handle when the current capacity is zero. In that case, we jump straight\nto eight elements instead of starting at one. That avoids a little extra memory churn when the array is very\nsmall, at the expense of wasting a few bytes on very small chunks.\n\n\n\nOnce we know the desired capacity, we create or grow the array to that size\nusing `GROW_ARRAY()`.\n\n^code grow-array (2 before, 2 after)\n\nThis macro pretties up a function call to `reallocate()` where the real work\nhappens. The macro itself takes care of getting the size of the array's element\ntype and casting the resulting `void*` back to a pointer of the right type.\n\nThis `reallocate()` function is the single function we'll use for all dynamic\nmemory management in clox -- allocating memory, freeing it, and changing the\nsize of an existing allocation. Routing all of those operations through a single\nfunction will be important later when we add a garbage collector that needs to\nkeep track of how much memory is in use.\n\nThe two size arguments passed to `reallocate()` control which operation to\nperform:\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
oldSizenewSizeOperation
0Non‑zeroAllocate new block.
Non‑zero0Free allocation.
Non‑zeroSmaller than oldSizeShrink existing allocation.
Non‑zeroLarger than oldSizeGrow existing allocation.
\n\nThat sounds like a lot of cases to handle, but here's the implementation:\n\n^code memory-c\n\nWhen `newSize` is zero, we handle the deallocation case ourselves by calling\n`free()`. Otherwise, we rely on the C standard library's `realloc()` function.\nThat function conveniently supports the other three aspects of our policy. When\n`oldSize` is zero, `realloc()` is equivalent to calling `malloc()`.\n\nThe interesting cases are when both `oldSize` and `newSize` are not zero. Those\ntell `realloc()` to resize the previously allocated block. If the new size is\nsmaller than the existing block of memory, it simply updates the size of the block and returns the same pointer\nyou gave it. If the new size is larger, it attempts to grow the existing block\nof memory.\n\nIt can do that only if the memory after that block isn't already in use. If\nthere isn't room to grow the block, `realloc()` instead allocates a *new* block\nof memory of the desired size, copies over the old bytes, frees the old block,\nand then returns a pointer to the new block. Remember, that's exactly the\nbehavior we want for our dynamic array.\n\nBecause computers are finite lumps of matter and not the perfect mathematical\nabstractions computer science theory would have us believe, allocation can fail\nif there isn't enough memory and `realloc()` will return `NULL`. We should\nhandle that.\n\n^code out-of-memory (1 before, 1 after)\n\nThere's not really anything *useful* that our VM can do if it can't get the\nmemory it needs, but we at least detect that and abort the process immediately\ninstead of returning a `NULL` pointer and letting it go off the rails later.\n\n\n\nOK, we can create new chunks and write instructions to them. Are we done? Nope!\nWe're in C now, remember, we have to manage memory ourselves, like in Ye Olden\nTimes, and that means *freeing* it too.\n\n^code free-chunk-h (1 before, 1 after)\n\nThe implementation is:\n\n^code free-chunk\n\nWe deallocate all of the memory and then call `initChunk()` to zero out the\nfields leaving the chunk in a well-defined empty state. To free the memory, we\nadd one more macro.\n\n^code free-array (3 before, 2 after)\n\nLike `GROW_ARRAY()`, this is a wrapper around a call to `reallocate()`. This one\nfrees the memory by passing in zero for the new size. I know, this is a lot of\nboring low-level stuff. Don't worry, we'll get a lot of use out of these in\nlater chapters and will get to program at a higher level. Before we can do that,\nthough, we gotta lay our own foundation.\n\n## Disassembling Chunks\n\nNow we have a little module for creating chunks of bytecode. Let's try it out by\nhand-building a sample chunk.\n\n^code main-chunk (1 before, 1 after)\n\nDon't forget the include.\n\n^code main-include-chunk (1 before, 2 after)\n\nRun that and give it a try. Did it work? Uh... who knows? All we've done is push\nsome bytes around in memory. We have no human-friendly way to see what's\nactually inside that chunk we made.\n\nTo fix this, we're going to create a **disassembler**. An **assembler** is an\nold-school program that takes a file containing human-readable mnemonic names\nfor CPU instructions like \"ADD\" and \"MULT\" and translates them to their binary\nmachine code equivalent. A *dis*assembler goes in the other direction -- given a\nblob of machine code, it spits out a textual listing of the instructions.\n\nWe'll implement something similar. Given a chunk, it\nwill print out all of the instructions in it. A Lox *user* won't use this, but\nwe Lox *maintainers* will certainly benefit since it gives us a window into the\ninterpreter's internal representation of code.\n\n\n\nIn `main()`, after we create the chunk, we pass it to the disassembler.\n\n^code main-disassemble-chunk (2 before, 1 after)\n\nAgain, we whip up yet another module.\n\n\n\n^code main-include-debug (1 before, 2 after)\n\nHere's that header:\n\n^code debug-h\n\nIn `main()`, we call `disassembleChunk()` to disassemble all of the instructions\nin the entire chunk. That's implemented in terms of the other function, which\njust disassembles a single instruction. It shows up here in the header because\nwe'll call it from the VM in later chapters.\n\nHere's a start at the implementation file:\n\n^code debug-c\n\nTo disassemble a chunk, we print a little header (so we can tell *which* chunk\nwe're looking at) and then crank through the bytecode, disassembling each\ninstruction. The way we iterate through the code is a little odd. Instead of\nincrementing `offset` in the loop, we let `disassembleInstruction()` do it for\nus. When we call that function, after disassembling the instruction at the given\noffset, it returns the offset of the *next* instruction. This is because, as\nwe'll see later, instructions can have different sizes.\n\nThe core of the \"debug\" module is this function:\n\n^code disassemble-instruction\n\nFirst, it prints the byte offset of the given instruction -- that tells us where\nin the chunk this instruction is. This will be a helpful signpost when we start\ndoing control flow and jumping around in the bytecode.\n\nNext, it reads a single byte from the bytecode at the given offset. That's our\nopcode. We switch on that. For each kind of\ninstruction, we dispatch to a little utility function for displaying it. On the\noff chance that the given byte doesn't look like an instruction at all -- a bug\nin our compiler -- we print that too. For the one instruction we do have,\n`OP_RETURN`, the display function is:\n\n\n\n^code simple-instruction\n\nThere isn't much to a return instruction, so all it does is print the name of\nthe opcode, then return the next byte offset past this instruction. Other\ninstructions will have more going on.\n\nIf we run our nascent interpreter now, it actually prints something:\n\n```text\n== test chunk ==\n0000 OP_RETURN\n```\n\nIt worked! This is sort of the \"Hello, world!\" of our code representation. We\ncan create a chunk, write an instruction to it, and then extract that\ninstruction back out. Our encoding and decoding of the binary bytecode is\nworking.\n\n## Constants\n\nNow that we have a rudimentary chunk structure working, let's start making it\nmore useful. We can store *code* in chunks, but what about *data*? Many values\nthe interpreter works with are created at runtime as the result of operations.\n\n```lox\n1 + 2;\n```\n\nThe value 3 appears nowhere in the code here. However, the literals `1` and `2`\ndo. To compile that statement to bytecode, we need some sort of instruction that\nmeans \"produce a constant\" and those literal values need to get stored in the\nchunk somewhere. In jlox, the Expr.Literal AST node held the value. We need a\ndifferent solution now that we don't have a syntax tree.\n\n### Representing values\n\nWe won't be *running* any code in this chapter, but since constants have a foot\nin both the static and dynamic worlds of our interpreter, they force us to start\nthinking at least a little bit about how our VM should represent values.\n\nFor now, we're going to start as simple as possible -- we'll support only\ndouble-precision, floating-point numbers. This will obviously expand over time,\nso we'll set up a new module to give ourselves room to grow.\n\n^code value-h\n\nThis typedef abstracts how Lox values are concretely represented in C. That way,\nwe can change that representation without needing to go back and fix existing\ncode that passes around values.\n\nBack to the question of where to store constants in a chunk. For small\nfixed-size values like integers, many instruction sets store the value directly\nin the code stream right after the opcode. These are called **immediate\ninstructions** because the bits for the value are immediately after the opcode.\n\nThat doesn't work well for large or variable-sized constants like strings. In a\nnative compiler to machine code, those bigger constants get stored in a separate\n\"constant data\" region in the binary executable. Then, the instruction to load a\nconstant has an address or offset pointing to where the value is stored in that\nsection.\n\nMost virtual machines do something similar. For example, the Java Virtual\nMachine [associates a **constant pool**][jvm const] with each compiled class.\nThat sounds good enough for clox to me. Each chunk will carry with it a list of\nthe values that appear as literals in the program. To keep things simpler, we'll put *all* constants in there, even simple\nintegers.\n\n[jvm const]: https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.4\n\n\n\n### Value arrays\n\nThe constant pool is an array of values. The instruction to load a constant\nlooks up the value by index in that array. As with our bytecode array, the compiler doesn't know how big the\narray needs to be ahead of time. So, again, we need a dynamic one. Since C\ndoesn't have generic data structures, we'll write another dynamic array data\nstructure, this time for Value.\n\n\n\n^code value-array (1 before, 2 after)\n\nAs with the bytecode array in Chunk, this struct wraps a pointer to an array\nalong with its allocated capacity and the number of elements in use. We also\nneed the same three functions to work with value arrays.\n\n^code array-fns-h (1 before, 2 after)\n\nThe implementations will probably give you déjà vu. First, to create a new one:\n\n^code value-c\n\nOnce we have an initialized array, we can start adding\nvalues to it.\n\n\n\n^code write-value-array\n\nThe memory-management macros we wrote earlier do let us reuse some of the logic\nfrom the code array, so this isn't too bad. Finally, to release all memory used\nby the array:\n\n^code free-value-array\n\nNow that we have growable arrays of values, we can add one to Chunk to store the\nchunk's constants.\n\n^code chunk-constants (1 before, 1 after)\n\nDon't forget the include.\n\n^code chunk-h-include-value (1 before, 2 after)\n\nAh, C, and its Stone Age modularity story. Where were we? Right. When we\ninitialize a new chunk, we initialize its constant list too.\n\n^code chunk-init-constant-array (1 before, 1 after)\n\nLikewise, we free the constants when we free the chunk.\n\n^code chunk-free-constants (1 before, 1 after)\n\nNext, we define a convenience method to add a new constant to the chunk. Our\nyet-to-be-written compiler could write to the constant array inside Chunk\ndirectly -- it's not like C has private fields or anything -- but it's a little\nnicer to add an explicit function.\n\n^code add-constant-h (1 before, 2 after)\n\nThen we implement it.\n\n^code add-constant\n\nAfter we add the constant, we return the index where the constant was appended\nso that we can locate that same constant later.\n\n### Constant instructions\n\nWe can *store* constants in chunks, but we also need to *execute* them. In a\npiece of code like:\n\n```lox\nprint 1;\nprint 2;\n```\n\nThe compiled chunk needs to not only contain the values 1 and 2, but know *when*\nto produce them so that they are printed in the right order. Thus, we need an\ninstruction that produces a particular constant.\n\n^code op-constant (1 before, 1 after)\n\nWhen the VM executes a constant instruction, it \"loads\"\nthe constant for use. This new instruction is a little more complex than\n`OP_RETURN`. In the above example, we load two different constants. A single\nbare opcode isn't enough to know *which* constant to load.\n\n\n\nTo handle cases like this, our bytecode -- like most others -- allows\ninstructions to have **operands**. These are stored\nas binary data immediately after the opcode in the instruction stream and let us\nparameterize what the instruction does.\n\n\"OP_CONSTANT\n\nEach opcode determines how many operand bytes it has and what they mean. For\nexample, a simple operation like \"return\" may have no operands, where an\ninstruction for \"load local variable\" needs an operand to identify which\nvariable to load. Each time we add a new opcode to clox, we specify what its\noperands look like -- its **instruction format**.\n\n\n\nIn this case, `OP_CONSTANT` takes a single byte operand that specifies which\nconstant to load from the chunk's constant array. Since we don't have a compiler\nyet, we \"hand-compile\" an instruction in our test chunk.\n\n^code main-constant (1 before, 1 after)\n\nWe add the constant value itself to the chunk's constant pool. That returns the\nindex of the constant in the array. Then we write the constant instruction,\nstarting with its opcode. After that, we write the one-byte constant index\noperand. Note that `writeChunk()` can write opcodes or operands. It's all raw\nbytes as far as that function is concerned.\n\nIf we try to run this now, the disassembler is going to yell at us because it\ndoesn't know how to decode the new instruction. Let's fix that.\n\n^code disassemble-constant (1 before, 1 after)\n\nThis instruction has a different instruction format, so we write a new helper\nfunction to disassemble it.\n\n^code constant-instruction\n\nThere's more going on here. As with `OP_RETURN`, we print out the name of the\nopcode. Then we pull out the constant index from the subsequent byte in the\nchunk. We print that index, but that isn't super useful to us human readers. So\nwe also look up the actual constant value -- since constants *are* known at\ncompile time after all -- and display the value itself too.\n\nThis requires some way to print a clox Value. That function will live in the\n\"value\" module, so we include that.\n\n^code debug-include-value (1 before, 2 after)\n\nOver in that header, we declare:\n\n^code print-value-h (1 before, 2 after)\n\nAnd here's an implementation:\n\n^code print-value\n\nMagnificent, right? As you can imagine, this is going to get more complex once\nwe add dynamic typing to Lox and have values of different types.\n\nBack in `constantInstruction()`, the only remaining piece is the return value.\n\n^code return-after-operand (1 before, 1 after)\n\nRemember that `disassembleInstruction()` also returns a number to tell the\ncaller the offset of the beginning of the *next* instruction. Where `OP_RETURN`\nwas only a single byte, `OP_CONSTANT` is two -- one for the opcode and one for\nthe operand.\n\n## Line Information\n\nChunks contain almost all of the information that the runtime needs from the\nuser's source code. It's kind of crazy to think that we can reduce all of the\ndifferent AST classes that we created in jlox down to an array of bytes and an\narray of constants. There's only one piece of data we're missing. We need it,\neven though the user hopes to never see it.\n\nWhen a runtime error occurs, we show the user the line number of the offending\nsource code. In jlox, those numbers live in tokens, which we in turn store in\nthe AST nodes. We need a different solution for clox now that we've ditched\nsyntax trees in favor of bytecode. Given any bytecode instruction, we need to be\nable to determine the line of the user's source program that it was compiled\nfrom.\n\nThere are a lot of clever ways we could encode this. I took the absolute simplest approach I could come up with, even though it's\nembarrassingly inefficient with memory. In the chunk, we store a separate array\nof integers that parallels the bytecode. Each number in the array is the line\nnumber for the corresponding byte in the bytecode. When a runtime error occurs,\nwe look up the line number at the same index as the current instruction's offset\nin the code array.\n\n\n\nTo implement this, we add another array to Chunk.\n\n^code chunk-lines (1 before, 1 after)\n\nSince it exactly parallels the bytecode array, we don't need a separate count or\ncapacity. Every time we touch the code array, we make a corresponding change to\nthe line number array, starting with initialization.\n\n^code chunk-null-lines (1 before, 1 after)\n\nAnd likewise deallocation:\n\n^code chunk-free-lines (1 before, 1 after)\n\nWhen we write a byte of code to the chunk, we need to know what source line it\ncame from, so we add an extra parameter in the declaration of `writeChunk()`.\n\n^code write-chunk-with-line-h (1 before, 1 after)\n\nAnd in the implementation:\n\n^code write-chunk-with-line (1 after)\n\nWhen we allocate or grow the code array, we do the same for the line info too.\n\n^code write-chunk-line (2 before, 1 after)\n\nFinally, we store the line number in the array.\n\n^code chunk-write-line (1 before, 1 after)\n\n### Disassembling line information\n\nAlright, let's try this out with our little, uh, artisanal chunk. First, since\nwe added a new parameter to `writeChunk()`, we need to fix those calls to pass\nin some -- arbitrary at this point -- line number.\n\n^code main-chunk-line (1 before, 2 after)\n\nOnce we have a real front end, of course, the compiler will track the current\nline as it parses and pass that in.\n\nNow that we have line information for every instruction, let's put it to good\nuse. In our disassembler, it's helpful to show which source line each\ninstruction was compiled from. That gives us a way to map back to the original\ncode when we're trying to figure out what some blob of bytecode is supposed to\ndo. After printing the offset of the instruction -- the number of bytes from the\nbeginning of the chunk -- we show its source line.\n\n^code show-location (2 before, 2 after)\n\nBytecode instructions tend to be pretty fine-grained. A single line of source\ncode often compiles to a whole sequence of instructions. To make that more\nvisually clear, we show a `|` for any instruction that comes from the same\nsource line as the preceding one. The resulting output for our handwritten\nchunk looks like:\n\n```text\n== test chunk ==\n0000 123 OP_CONSTANT 0 '1.2'\n0002 | OP_RETURN\n```\n\nWe have a three-byte chunk. The first two bytes are a constant instruction that\nloads 1.2 from the chunk's constant pool. The first byte is the `OP_CONSTANT`\nopcode and the second is the index in the constant pool. The third byte (at\noffset 2) is a single-byte return instruction.\n\nIn the remaining chapters, we will flesh this out with lots more kinds of\ninstructions. But the basic structure is here, and we have everything we need\nnow to completely represent an executable piece of code at runtime in our\nvirtual machine. Remember that whole family of AST classes we defined in jlox?\nIn clox, we've reduced that down to three arrays: bytes of code, constant\nvalues, and line information for debugging.\n\nThis reduction is a key reason why our new interpreter will be faster than jlox.\nYou can think of bytecode as a sort of compact serialization of the AST, highly\noptimized for how the interpreter will deserialize it in the order it needs as\nit executes. In the [next chapter][vm], we will see how the virtual machine does\nexactly that.\n\n
\n\n## Challenges\n\n1. Our encoding of line information is hilariously wasteful of memory. Given\n that a series of instructions often correspond to the same source line, a\n natural solution is something akin to [run-length encoding][rle] of the line\n numbers.\n\n Devise an encoding that compresses the line information for a\n series of instructions on the same line. Change `writeChunk()` to write this\n compressed form, and implement a `getLine()` function that, given the index\n of an instruction, determines the line where the instruction occurs.\n\n *Hint: It's not necessary for `getLine()` to be particularly efficient.\n Since it is called only when a runtime error occurs, it is well off the\n critical path where performance matters.*\n\n2. Because `OP_CONSTANT` uses only a single byte for its operand, a chunk may\n only contain up to 256 different constants. That's small enough that people\n writing real-world code will hit that limit. We could use two or more bytes\n to store the operand, but that makes *every* constant instruction take up\n more space. Most chunks won't need that many unique constants, so that\n wastes space and sacrifices some locality in the common case to support the\n rare case.\n\n To balance those two competing aims, many instruction sets feature multiple\n instructions that perform the same operation but with operands of different\n sizes. Leave our existing one-byte `OP_CONSTANT` instruction alone, and\n define a second `OP_CONSTANT_LONG` instruction. It stores the operand as a\n 24-bit number, which should be plenty.\n\n Implement this function:\n\n ```c\n void writeConstant(Chunk* chunk, Value value, int line) {\n // Implement me...\n }\n ```\n\n It adds `value` to `chunk`'s constant array and then writes an appropriate\n instruction to load the constant. Also add support to the disassembler for\n `OP_CONSTANT_LONG` instructions.\n\n Defining two instructions seems to be the best of both worlds. What\n sacrifices, if any, does it force on us?\n\n3. Our `reallocate()` function relies on the C standard library for dynamic\n memory allocation and freeing. `malloc()` and `free()` aren't magic. Find\n a couple of open source implementations of them and explain how they work.\n How do they keep track of which bytes are allocated and which are free?\n What is required to allocate a block of memory? Free it? How do they make\n that efficient? What do they do about fragmentation?\n\n *Hardcore mode:* Implement `reallocate()` without calling `realloc()`,\n `malloc()`, or `free()`. You are allowed to call `malloc()` *once*, at the\n beginning of the interpreter's execution, to allocate a single big block of\n memory, which your `reallocate()` function has access to. It parcels out\n blobs of memory from that single region, your own personal heap. It's your\n job to define how it does that.\n\n
\n\n[rle]: https://en.wikipedia.org/wiki/Run-length_encoding\n\n
\n\n## Design Note: Test Your Language\n\nWe're almost halfway through the book and one thing we haven't talked about is\n*testing* your language implementation. That's not because testing isn't\nimportant. I can't possibly stress enough how vital it is to have a good,\ncomprehensive test suite for your language.\n\nI wrote a [test suite for Lox][tests] (which you are welcome to use on your own\nLox implementation) before I wrote a single word of this book. Those tests found\ncountless bugs in my implementations.\n\n[tests]: https://github.com/munificent/craftinginterpreters/tree/master/test\n\nTests are important in all software, but they're even more important for a\nprogramming language for at least a couple of reasons:\n\n* **Users expect their programming languages to be rock solid.** We are so\n used to mature, stable compilers and interpreters that \"It's your code, not\n the compiler\" is [an ingrained part of software culture][fault]. If there\n are bugs in your language implementation, users will go through the full\n five stages of grief before they can figure out what's going on, and you\n don't want to put them through all that.\n\n* **A language implementation is a deeply interconnected piece of software.**\n Some codebases are broad and shallow. If the file loading code is broken in\n your text editor, it -- hopefully! -- won't cause failures in the text\n rendering on screen. Language implementations are narrower and deeper,\n especially the core of the interpreter that handles the language's actual\n semantics. That makes it easy for subtle bugs to creep in caused by weird\n interactions between various parts of the system. It takes good tests to\n flush those out.\n\n* **The input to a language implementation is, by design, combinatorial.**\n There are an infinite number of possible programs a user could write, and\n your implementation needs to run them all correctly. You obviously can't\n test that exhaustively, but you need to work hard to cover as much of the\n input space as you can.\n\n* **Language implementations are often complex, constantly changing, and full\n of optimizations.** That leads to gnarly code with lots of dark corners\n where bugs can hide.\n\n[fault]: https://blog.codinghorror.com/the-first-rule-of-programming-its-always-your-fault/\n\nAll of that means you're gonna want a lot of tests. But *what* tests? Projects\nI've seen focus mostly on end-to-end \"language tests\". Each test is a program\nwritten in the language along with the output or errors it is expected to\nproduce. Then you have a test runner that pushes the test program through your\nlanguage implementation and validates that it does what it's supposed to.\nWriting your tests in the language itself has a few nice advantages:\n\n* The tests aren't coupled to any particular API or internal architecture\n decisions of the implementation. This frees you to reorganize or rewrite\n parts of your interpreter or compiler without needing to update a slew of\n tests.\n\n* You can use the same tests for multiple implementations of the language.\n\n* Tests can often be terse and easy to read and maintain since they are\n simply scripts in your language.\n\nIt's not all rosy, though:\n\n* End-to-end tests help you determine *if* there is a bug, but not *where* the\n bug is. It can be harder to figure out where the erroneous code in the\n implementation is because all the test tells you is that the right output\n didn't appear.\n\n* It can be a chore to craft a valid program that tickles some obscure corner\n of the implementation. This is particularly true for highly optimized\n compilers where you may need to write convoluted code to ensure that you\n end up on just the right optimization path where a bug may be hiding.\n\n* The overhead can be high to fire up the interpreter, parse, compile, and\n run each test script. With a big suite of tests -- which you *do* want,\n remember -- that can mean a lot of time spent waiting for the tests to\n finish running.\n\nI could go on, but I don't want this to turn into a sermon. Also, I don't\npretend to be an expert on *how* to test languages. I just want you to\ninternalize how important it is *that* you test yours. Seriously. Test your\nlanguage. You'll thank me for it.\n\n
\n" }, { "path": "book/classes-and-instances.md", "content": "> Caring too much for objects can destroy you. Only -- if you care for a thing\n> enough, it takes on a life of its own, doesn't it? And isn’t the whole point\n> of things -- beautiful things -- that they connect you to some larger beauty?\n>\n> Donna Tartt, The Goldfinch\n\nThe last area left to implement in clox is object-oriented programming. OOP is a bundle of intertwined features: classes, instances,\nfields, methods, initializers, and inheritance. Using relatively high-level\nJava, we packed all that into two chapters. Now that we're coding in C, which\nfeels like building a model of the Eiffel tower out of toothpicks, we'll devote\nthree chapters to covering the same territory. This makes for a leisurely stroll\nthrough the implementation. After strenuous chapters like [closures][] and the\n[garbage collector][], you have earned a rest. In fact, the book should be easy\nfrom here on out.\n\n\n\nIn this chapter, we cover the first three features: classes, instances, and\nfields. This is the stateful side of object orientation. Then in the next two\nchapters, we will hang behavior and code reuse off of those objects.\n\n[closures]: closures.html\n[garbage collector]: garbage-collection.html\n\n## Class Objects\n\nIn a class-based object-oriented language, everything begins with classes. They\ndefine what sorts of objects exist in the program and are the factories used to\nproduce new instances. Going bottom-up, we'll start with their runtime\nrepresentation and then hook that into the language.\n\nBy this point, we're well-acquainted with the process of adding a new object\ntype to the VM. We start with a struct.\n\n^code obj-class (1 before, 2 after)\n\nAfter the Obj header, we store the class's name. This isn't strictly needed for\nthe user's program, but it lets us show the name at runtime for things like\nstack traces.\n\nThe new type needs a corresponding case in the ObjType enum.\n\n^code obj-type-class (1 before, 1 after)\n\nAnd that type gets a corresponding pair of macros. First, for testing an\nobject's type:\n\n^code is-class (2 before, 1 after)\n\nAnd then for casting a Value to an ObjClass pointer:\n\n^code as-class (2 before, 1 after)\n\nThe VM creates new class objects using this function:\n\n^code new-class-h (2 before, 1 after)\n\nThe implementation lives over here:\n\n^code new-class\n\nPretty much all boilerplate. It takes in the class's name as a string and stores\nit. Every time the user declares a new class, the VM will create a new one of\nthese ObjClass structs to represent it.\n\n\n\nWhen the VM no longer needs a class, it frees it like so:\n\n^code free-class (1 before, 1 after)\n\n\n\nWe have a memory manager now, so we also need to support tracing through class\nobjects.\n\n^code blacken-class (1 before, 1 after)\n\nWhen the GC reaches a class object, it marks the class's name to keep that\nstring alive too.\n\nThe last operation the VM can perform on a class is printing it.\n\n^code print-class (1 before, 1 after)\n\nA class simply says its own name.\n\n## Class Declarations\n\nRuntime representation in hand, we are ready to add support for classes to the\nlanguage. Next, we move into the parser.\n\n^code match-class (1 before, 1 after)\n\nClass declarations are statements, and the parser recognizes one by the leading\n`class` keyword. The rest of the compilation happens over here:\n\n^code class-declaration\n\nImmediately after the `class` keyword is the class's name. We take that\nidentifier and add it to the surrounding function's constant table as a string.\nAs you just saw, printing a class shows its name, so the compiler needs to stuff\nthe name string somewhere that the runtime can find. The constant table is the\nway to do that.\n\nThe class's name is also used to bind the class\nobject to a variable of the same name. So we declare a variable with that\nidentifier right after consuming its token.\n\n\n\nNext, we emit a new instruction to actually create the class object at runtime.\nThat instruction takes the constant table index of the class's name as an\noperand.\n\nAfter that, but before compiling the body of the class, we define the variable\nfor the class's name. *Declaring* the variable adds it to the scope, but recall\nfrom [a previous chapter][scope] that we can't *use* the variable until it's\n*defined*. For classes, we define the variable before the body. That way, users\ncan refer to the containing class inside the bodies of its own methods. That's\nuseful for things like factory methods that produce new instances of the class.\n\n[scope]: local-variables.html#another-scope-edge-case\n\nFinally, we compile the body. We don't have methods yet, so right now it's\nsimply an empty pair of braces. Lox doesn't require fields to be declared in the\nclass, so we're done with the body -- and the parser -- for now.\n\nThe compiler is emitting a new instruction, so let's define that.\n\n^code class-op (1 before, 1 after)\n\nAnd add it to the disassembler:\n\n^code disassemble-class (2 before, 1 after)\n\nFor such a large-seeming feature, the interpreter support is minimal.\n\n^code interpret-class (2 before, 1 after)\n\nWe load the string for the class's name from the constant table and pass that to\n`newClass()`. That creates a new class object with the given name. We push that\nonto the stack and we're good. If the class is bound to a global variable, then\nthe compiler's call to `defineVariable()` will emit code to store that object\nfrom the stack into the global variable table. Otherwise, it's right where it\nneeds to be on the stack for a new local variable.\n\n\n\nThere you have it, our VM supports classes now. You can run this:\n\n```lox\nclass Brioche {}\nprint Brioche;\n```\n\nUnfortunately, printing is about *all* you can do with classes, so next is\nmaking them more useful.\n\n## Instances of Classes\n\nClasses serve two main purposes in a language:\n\n* **They are how you create new instances.** Sometimes this involves a `new`\n keyword, other times it's a method call on the class object, but you usually\n mention the class by name *somehow* to get a new instance.\n\n* **They contain methods.** These define how all instances of the class\n behave.\n\nWe won't get to methods until the next chapter, so for now we will only worry\nabout the first part. Before classes can create instances, we need a\nrepresentation for them.\n\n^code obj-instance (1 before, 2 after)\n\nInstances know their class -- each instance has a pointer to the class that it\nis an instance of. We won't use this much in this chapter, but it will become\ncritical when we add methods.\n\nMore important to this chapter is how instances store their state. Lox lets\nusers freely add fields to an instance at runtime. This means we need a storage\nmechanism that can grow. We could use a dynamic array, but we also want to look\nup fields by name as quickly as possible. There's a data structure that's just\nperfect for quickly accessing a set of values by name and\n-- even more conveniently -- we've already implemented it. Each instance stores\nits fields using a hash table.\n\n\n\nWe only need to add an include, and we've got it.\n\n^code object-include-table (1 before, 1 after)\n\nThis new struct gets a new object type.\n\n^code obj-type-instance (1 before, 1 after)\n\nI want to slow down a bit here because the Lox *language's* notion of \"type\" and\nthe VM *implementation's* notion of \"type\" brush against each other in ways that\ncan be confusing. Inside the C code that makes clox, there are a number of\ndifferent types of Obj -- ObjString, ObjClosure, etc. Each has its own internal\nrepresentation and semantics.\n\nIn the Lox *language*, users can define their own classes -- say Cake and Pie --\nand then create instances of those classes. From the user's perspective, an\ninstance of Cake is a different type of object than an instance of Pie. But,\nfrom the VM's perspective, every class the user defines is simply another value\nof type ObjClass. Likewise, each instance in the user's program, no matter what\nclass it is an instance of, is an ObjInstance. That one VM object type covers\ninstances of all classes. The two worlds map to each other something like this:\n\n\"A\n\nGot it? OK, back to the implementation. We also get our usual macros.\n\n^code is-instance (1 before, 1 after)\n\nAnd:\n\n^code as-instance (1 before, 1 after)\n\nSince fields are added after the instance is created, the \"constructor\" function\nonly needs to know the class.\n\n^code new-instance-h (1 before, 1 after)\n\nWe implement that function here:\n\n^code new-instance\n\nWe store a reference to the instance's class. Then we initialize the field\ntable to an empty hash table. A new baby object is born!\n\nAt the sadder end of the instance's lifespan, it gets freed.\n\n^code free-instance (3 before, 1 after)\n\nThe instance owns its field table so when freeing the instance, we also free the\ntable. We don't explicitly free the entries *in* the table, because there may\nbe other references to those objects. The garbage collector will take care of\nthose for us. Here we free only the entry array of the table itself.\n\nSpeaking of the garbage collector, it needs support for tracing through\ninstances.\n\n^code blacken-instance (3 before, 1 after)\n\nIf the instance is alive, we need to keep its class around. Also, we need to\nkeep every object referenced by the instance's fields. Most live objects that\nare not roots are reachable because some instance refers to the object in a\nfield. Fortunately, we already have a nice `markTable()` function to make\ntracing them easy.\n\nLess critical but still important is printing.\n\n^code print-instance (1 before, 1 after)\n\nAn instance prints its name followed by \"instance\".\n(The \"instance\" part is mainly so that classes and instances don't print the\nsame.)\n\n\n\nThe real fun happens over in the interpreter. Lox has no special `new` keyword.\nThe way to create an instance of a class is to invoke the class itself as if it\nwere a function. The runtime already supports function calls, and it checks the\ntype of object being called to make sure the user doesn't try to invoke a number\nor other invalid type.\n\nWe extend that runtime checking with a new case.\n\n^code call-class (1 before, 1 after)\n\nIf the value being called -- the object that results when evaluating the\nexpression to the left of the opening parenthesis -- is a class, then we treat\nit as a constructor call. We create a new instance of\nthe called class and store the result on the stack.\n\n\n\nWe're one step farther. Now we can define classes and create instances of them.\n\n```lox\nclass Brioche {}\nprint Brioche();\n```\n\nNote the parentheses after `Brioche` on the second line now. This prints\n\"Brioche instance\".\n\n## Get and Set Expressions\n\nOur object representation for instances can already store state, so all that\nremains is exposing that functionality to the user. Fields are accessed and\nmodified using get and set expressions. Not one to break with tradition, Lox\nuses the classic \"dot\" syntax:\n\n```lox\neclair.filling = \"pastry creme\";\nprint eclair.filling;\n```\n\nThe period -- full stop for my English friends -- works sort of like an infix operator. There is an expression to the\nleft that is evaluated first and produces an instance. After that is the `.`\nfollowed by a field name. Since there is a preceding operand, we hook this into\nthe parse table as an infix expression.\n\n\n\n^code table-dot (1 before, 1 after)\n\nAs in other languages, the `.` operator binds tightly, with precedence as high\nas the parentheses in a function call. After the parser consumes the dot token,\nit dispatches to a new parse function.\n\n^code compile-dot\n\nThe parser expects to find a property name immediately\nafter the dot. We load that token's lexeme into the constant table as a string\nso that the name is available at runtime.\n\n\n\nWe have two new expression forms -- getters and setters -- that this one\nfunction handles. If we see an equals sign after the field name, it must be a\nset expression that is assigning to a field. But we don't *always* allow an\nequals sign after the field to be compiled. Consider:\n\n```lox\na + b.c = 3\n```\n\nThis is syntactically invalid according to Lox's grammar, which means our Lox\nimplementation is obligated to detect and report the error. If `dot()` silently\nparsed the `= 3` part, we would incorrectly interpret the code as if the user\nhad written:\n\n```lox\na + (b.c = 3)\n```\n\nThe problem is that the `=` side of a set expression has much lower precedence\nthan the `.` part. The parser may call `dot()` in a context that is too high\nprecedence to permit a setter to appear. To avoid incorrectly allowing that, we\nparse and compile the equals part only when `canAssign` is true. If an equals\ntoken appears when `canAssign` is false, `dot()` leaves it alone and returns. In\nthat case, the compiler will eventually unwind up to `parsePrecedence()`, which\nstops at the unexpected `=` still sitting as the next token and reports an\nerror.\n\nIf we find an `=` in a context where it *is* allowed, then we compile the\nexpression that follows. After that, we emit a new `OP_SET_PROPERTY` instruction. That takes a single operand for\nthe index of the property name in the constant table. If we didn't compile a set\nexpression, we assume it's a getter and emit an `OP_GET_PROPERTY` instruction,\nwhich also takes an operand for the property name.\n\n\n\nNow is a good time to define these two new instructions.\n\n^code property-ops (1 before, 1 after)\n\nAnd add support for disassembling them:\n\n^code disassemble-property-ops (1 before, 1 after)\n\n### Interpreting getter and setter expressions\n\nSliding over to the runtime, we'll start with get expressions since those are a\nlittle simpler.\n\n^code interpret-get-property (1 before, 1 after)\n\nWhen the interpreter reaches this instruction, the expression to the left of the\ndot has already been executed and the resulting instance is on top of the stack.\nWe read the field name from the constant pool and look it up in the instance's\nfield table. If the hash table contains an entry with that name, we pop the\ninstance and push the entry's value as the result.\n\nOf course, the field might not exist. In Lox, we've defined that to be a runtime\nerror. So we add a check for that and abort if it happens.\n\n^code get-undefined (3 before, 2 after)\n\nThere is another failure mode to handle which you've\nprobably noticed. The above code assumes the expression to the left of the dot\ndid evaluate to an ObjInstance. But there's nothing preventing a user from\nwriting this:\n\n```lox\nvar obj = \"not an instance\";\nprint obj.field;\n```\n\nThe user's program is wrong, but the VM still has to handle it with some grace.\nRight now, it will misinterpret the bits of the ObjString as an ObjInstance and,\nI don't know, catch on fire or something definitely not graceful.\n\nIn Lox, only instances are allowed to have fields. You can't stuff a field onto\na string or number. So we need to check that the value is an instance before\naccessing any fields on it.\n\n\n\n^code get-not-instance (1 before, 1 after)\n\nIf the value on the stack isn't an instance, we report a runtime error and\nsafely exit.\n\nOf course, get expressions are not very useful when no instances have any\nfields. For that we need setters.\n\n^code interpret-set-property (2 before, 1 after)\n\nThis is a little more complex than `OP_GET_PROPERTY`. When this executes, the\ntop of the stack has the instance whose field is being set and above that, the\nvalue to be stored. Like before, we read the instruction's operand and find the\nfield name string. Using that, we store the value on top of the stack into the\ninstance's field table.\n\nAfter that is a little stack juggling. We pop the\nstored value off, then pop the instance, and finally push the value back on. In\nother words, we remove the *second* element from the stack while leaving the top\nalone. A setter is itself an expression whose result is the assigned value, so\nwe need to leave that value on the stack. Here's what I mean:\n\n\n\n```lox\nclass Toast {}\nvar toast = Toast();\nprint toast.jam = \"grape\"; // Prints \"grape\".\n```\n\nUnlike when reading a field, we don't need to worry about the hash table not\ncontaining the field. A setter implicitly creates the field if needed. We do\nneed to handle the user incorrectly trying to store a field on a value that\nisn't an instance.\n\n^code set-not-instance (1 before, 1 after)\n\nExactly like with get expressions, we check the value's type and report a\nruntime error if it's invalid. And, with that, the stateful side of Lox's\nsupport for object-oriented programming is in place. Give it a try:\n\n```lox\nclass Pair {}\n\nvar pair = Pair();\npair.first = 1;\npair.second = 2;\nprint pair.first + pair.second; // 3.\n```\n\nThis doesn't really feel very *object*-oriented. It's more like a strange,\ndynamically typed variant of C where objects are loose struct-like bags of data.\nSort of a dynamic procedural language. But this is a big step in expressiveness.\nOur Lox implementation now lets users freely aggregate data into bigger units.\nIn the next chapter, we will breathe life into those inert blobs.\n\n
\n\n## Challenges\n\n1. Trying to access a non-existent field on an object immediately aborts the\n entire VM. The user has no way to recover from this runtime error, nor is\n there any way to see if a field exists *before* trying to access it. It's up\n to the user to ensure on their own that only valid fields are read.\n\n How do other dynamically typed languages handle missing fields? What do you\n think Lox should do? Implement your solution.\n\n2. Fields are accessed at runtime by their *string* name. But that name must\n always appear directly in the source code as an *identifier token*. A user\n program cannot imperatively build a string value and then use that as the\n name of a field. Do you think they should be able to? Devise a language\n feature that enables that and implement it.\n\n3. Conversely, Lox offers no way to *remove* a field from an instance. You can\n set a field's value to `nil`, but the entry in the hash table is still\n there. How do other languages handle this? Choose and implement a strategy\n for Lox.\n\n4. Because fields are accessed by name at runtime, working with instance state\n is slow. It's technically a constant-time operation -- thanks, hash tables\n -- but the constant factors are relatively large. This is a major component\n of why dynamic languages are slower than statically typed ones.\n\n How do sophisticated implementations of dynamically typed languages cope\n with and optimize this?\n\n
\n" }, { "path": "book/classes.md", "content": "> One has no right to love or hate anything if one has not acquired a thorough\n> knowledge of its nature. Great love springs from great knowledge of the\n> beloved object, and if you know it but little you will be able to love it only\n> a little or not at all.\n>\n> Leonardo da Vinci\n\nWe're eleven chapters in, and the interpreter sitting on your machine is nearly\na complete scripting language. It could use a couple of built-in data structures\nlike lists and maps, and it certainly needs a core library for file I/O, user\ninput, etc. But the language itself is sufficient. We've got a little procedural\nlanguage in the same vein as BASIC, Tcl, Scheme (minus macros), and early\nversions of Python and Lua.\n\nIf this were the '80s, we'd stop here. But today, many popular languages support\n\"object-oriented programming\". Adding that to Lox will give users a familiar set\nof tools for writing larger programs. Even if you personally don't like OOP, this chapter and [the next][inheritance] will help\nyou understand how others design and build object systems.\n\n[inheritance]: inheritance.html\n\n\n\n## OOP and Classes\n\nThere are three broad paths to object-oriented programming: classes,\n[prototypes][], and [multimethods][]. Classes\ncame first and are the most popular style. With the rise of JavaScript (and to a\nlesser extent [Lua][]), prototypes are more widely known than they used to be.\nI'll talk more about those [later][]. For Lox, we're taking the, ahem, classic\napproach.\n\n[prototypes]: http://gameprogrammingpatterns.com/prototype.html\n[multimethods]: https://en.wikipedia.org/wiki/Multiple_dispatch\n[lua]: https://www.lua.org/pil/13.4.1.html\n[later]: #design-note\n\n\n\nSince you've written about a thousand lines of Java code with me already, I'm\nassuming you don't need a detailed introduction to object orientation. The main\ngoal is to bundle data with the code that acts on it. Users do that by declaring\na *class* that:\n\n\n\n1. Exposes a *constructor* to create and initialize new *instances* of the\n class\n\n1. Provides a way to store and access *fields* on instances\n\n1. Defines a set of *methods* shared by all instances of the class that\n operate on each instances' state.\n\nThat's about as minimal as it gets. Most object-oriented languages, all the way\nback to Simula, also do inheritance to reuse behavior across classes. We'll add\nthat in the [next chapter][inheritance]. Even kicking that out, we still have a\nlot to get through. This is a big chapter and everything doesn't quite come\ntogether until we have all of the above pieces, so gather your stamina.\n\n\n\n[inheritance]: inheritance.html\n\n## Class Declarations\n\nLike we do, we're gonna start with syntax. A `class` statement introduces a new\nname, so it lives in the `declaration` grammar rule.\n\n```ebnf\ndeclaration → classDecl\n | funDecl\n | varDecl\n | statement ;\n\nclassDecl → \"class\" IDENTIFIER \"{\" function* \"}\" ;\n```\n\nThe new `classDecl` rule relies on the `function` rule we defined\n[earlier][function rule]. To refresh your memory:\n\n[function rule]: functions.html#function-declarations\n\n```ebnf\nfunction → IDENTIFIER \"(\" parameters? \")\" block ;\nparameters → IDENTIFIER ( \",\" IDENTIFIER )* ;\n```\n\nIn plain English, a class declaration is the `class` keyword, followed by the\nclass's name, then a curly-braced body. Inside that body is a list of method\ndeclarations. Unlike function declarations, methods don't have a leading `fun` keyword. Each method is a name, parameter list, and\nbody. Here's an example:\n\n\n\n```lox\nclass Breakfast {\n cook() {\n print \"Eggs a-fryin'!\";\n }\n\n serve(who) {\n print \"Enjoy your breakfast, \" + who + \".\";\n }\n}\n```\n\nLike most dynamically typed languages, fields are not explicitly listed in the\nclass declaration. Instances are loose bags of data and you can freely add\nfields to them as you see fit using normal imperative code.\n\nOver in our AST generator, the `classDecl` grammar rule gets its own statement\nnode.\n\n^code class-ast (1 before, 1 after)\n\n\n\nIt stores the class's name and the methods inside its body. Methods are\nrepresented by the existing Stmt.Function class that we use for function\ndeclaration AST nodes. That gives us all the bits of state that we need for a\nmethod: name, parameter list, and body.\n\nA class can appear anywhere a named declaration is allowed, triggered by the\nleading `class` keyword.\n\n^code match-class (1 before, 1 after)\n\nThat calls out to:\n\n^code parse-class-declaration\n\nThere's more meat to this than most of the other parsing methods, but it roughly\nfollows the grammar. We've already consumed the `class` keyword, so we look for\nthe expected class name next, followed by the opening curly brace. Once inside\nthe body, we keep parsing method declarations until we hit the closing brace.\nEach method declaration is parsed by a call to `function()`, which we defined\nback in the [chapter where functions were introduced][functions].\n\n[functions]: functions.html\n\nLike we do in any open-ended loop in the parser, we also check for hitting the\nend of the file. That won't happen in correct code since a class should have a\nclosing brace at the end, but it ensures the parser doesn't get stuck in an\ninfinite loop if the user has a syntax error and forgets to correctly end the\nclass body.\n\nWe wrap the name and list of methods into a Stmt.Class node and we're done.\nPreviously, we would jump straight into the interpreter, but now we need to\nplumb the node through the resolver first.\n\n^code resolver-visit-class\n\nWe aren't going to worry about resolving the methods themselves yet, so for now\nall we need to do is declare the class using its name. It's not common to\ndeclare a class as a local variable, but Lox permits it, so we need to handle it\ncorrectly.\n\nNow we interpret the class declaration.\n\n^code interpreter-visit-class\n\nThis looks similar to how we execute function declarations. We declare the\nclass's name in the current environment. Then we turn the class *syntax node*\ninto a LoxClass, the *runtime* representation of a class. We circle back and\nstore the class object in the variable we previously declared. That two-stage\nvariable binding process allows references to the class inside its own methods.\n\nWe will refine it throughout the chapter, but the first draft of LoxClass looks\nlike this:\n\n^code lox-class\n\nLiterally a wrapper around a name. We don't even store the methods yet. Not\nsuper useful, but it does have a `toString()` method so we can write a trivial\nscript and test that class objects are actually being parsed and executed.\n\n```lox\nclass DevonshireCream {\n serveOn() {\n return \"Scones\";\n }\n}\n\nprint DevonshireCream; // Prints \"DevonshireCream\".\n```\n\n## Creating Instances\n\nWe have classes, but they don't do anything yet. Lox doesn't have \"static\"\nmethods that you can call right on the class itself, so without actual\ninstances, classes are useless. Thus instances are the next step.\n\nWhile some syntax and semantics are fairly standard across OOP languages, the\nway you create new instances isn't. Ruby, following Smalltalk, creates instances\nby calling a method on the class object itself, a recursively graceful approach. Some, like C++ and Java,\nhave a `new` keyword dedicated to birthing a new object. Python has you \"call\"\nthe class itself like a function. (JavaScript, ever weird, sort of does both.)\n\n\n\nI took a minimal approach with Lox. We already have class objects, and we\nalready have function calls, so we'll use call expressions on class objects to\ncreate new instances. It's as if a class is a factory function that generates\ninstances of itself. This feels elegant to me, and also spares us the need to\nintroduce syntax like `new`. Therefore, we can skip past the front end straight\ninto the runtime.\n\nRight now, if you try this:\n\n```lox\nclass Bagel {}\nBagel();\n```\n\nYou get a runtime error. `visitCallExpr()` checks to see if the called object\nimplements `LoxCallable` and reports an error since LoxClass doesn't. Not *yet*,\nthat is.\n\n^code lox-class-callable (2 before, 1 after)\n\nImplementing that interface requires two methods.\n\n^code lox-class-call-arity\n\nThe interesting one is `call()`. When you \"call\" a class, it instantiates a new\nLoxInstance for the called class and returns it. The `arity()` method is how the\ninterpreter validates that you passed the right number of arguments to a\ncallable. For now, we'll say you can't pass any. When we get to user-defined\nconstructors, we'll revisit this.\n\nThat leads us to LoxInstance, the runtime representation of an instance of a Lox\nclass. Again, our first implementation starts small.\n\n^code lox-instance\n\nLike LoxClass, it's pretty bare bones, but we're only getting started. If you\nwant to give it a try, here's a script to run:\n\n```lox\nclass Bagel {}\nvar bagel = Bagel();\nprint bagel; // Prints \"Bagel instance\".\n```\n\nThis program doesn't do much, but it's starting to do *something*.\n\n## Properties on Instances\n\nWe have instances, so we should make them useful. We're at a fork in the road.\nWe could add behavior first -- methods -- or we could start with state --\nproperties. We're going to take the latter because, as we'll see, the two get\nentangled in an interesting way and it will be easier to make sense of them if\nwe get properties working first.\n\nLox follows JavaScript and Python in how it handles state. Every instance is an\nopen collection of named values. Methods on the instance's class can access and\nmodify properties, but so can outside code.\nProperties are accessed using a `.` syntax.\n\n\n\n```lox\nsomeObject.someProperty\n```\n\nAn expression followed by `.` and an identifier reads the property with that\nname from the object the expression evaluates to. That dot has the same\nprecedence as the parentheses in a function call expression, so we slot it into\nthe grammar by replacing the existing `call` rule with:\n\n```ebnf\ncall → primary ( \"(\" arguments? \")\" | \".\" IDENTIFIER )* ;\n```\n\nAfter a primary expression, we allow a series of any mixture of parenthesized\ncalls and dotted property accesses. \"Property access\" is a mouthful, so from\nhere on out, we'll call these \"get expressions\".\n\n### Get expressions\n\nThe syntax tree node is:\n\n^code get-ast (1 before, 1 after)\n\n\n\nFollowing the grammar, the new parsing code goes in our existing `call()`\nmethod.\n\n^code parse-property (3 before, 4 after)\n\nThe outer `while` loop there corresponds to the `*` in the grammar rule. We zip\nalong the tokens building up a chain of calls and gets as we find parentheses\nand dots, like so:\n\n\"Parsing\n\nInstances of the new Expr.Get node feed into the resolver.\n\n^code resolver-visit-get\n\nOK, not much to that. Since properties are looked up dynamically, they don't get resolved. During resolution,\nwe recurse only into the expression to the left of the dot. The actual property\naccess happens in the interpreter.\n\n\n\n^code interpreter-visit-get\n\nFirst, we evaluate the expression whose property is being accessed. In Lox, only\ninstances of classes have properties. If the object is some other type like a\nnumber, invoking a getter on it is a runtime error.\n\nIf the object is a LoxInstance, then we ask it to look up the property. It must\nbe time to give LoxInstance some actual state. A map will do fine.\n\n^code lox-instance-fields (1 before, 2 after)\n\nEach key in the map is a property name and the corresponding value is the\nproperty's value. To look up a property on an instance:\n\n^code lox-instance-get-property\n\n\n\nAn interesting edge case we need to handle is what happens if the instance\ndoesn't *have* a property with the given name. We could silently return some\ndummy value like `nil`, but my experience with languages like JavaScript is that\nthis behavior masks bugs more often than it does anything useful. Instead, we'll\nmake it a runtime error.\n\nSo the first thing we do is see if the instance actually has a field with the\ngiven name. Only then do we return it. Otherwise, we raise an error.\n\nNote how I switched from talking about \"properties\" to \"fields\". There is a\nsubtle difference between the two. Fields are named bits of state stored\ndirectly in an instance. Properties are the named, uh, *things*, that a get\nexpression may return. Every field is a property, but as we'll see later, not every property is a field.\n\n\n\nIn theory, we can now read properties on objects. But since there's no way to\nactually stuff any state into an instance, there are no fields to access. Before\nwe can test out reading, we must support writing.\n\n### Set expressions\n\nSetters use the same syntax as getters, except they appear on the left side of\nan assignment.\n\n```lox\nsomeObject.someProperty = value;\n```\n\nIn grammar land, we extend the rule for assignment to allow dotted identifiers\non the left-hand side.\n\n```ebnf\nassignment → ( call \".\" )? IDENTIFIER \"=\" assignment\n | logic_or ;\n```\n\nUnlike getters, setters don't chain. However, the reference to `call` allows any\nhigh-precedence expression before the last dot, including any number of\n*getters*, as in:\n\n\"breakfast.omelette.filling.meat\n\nNote here that only the *last* part, the `.meat` is the *setter*. The\n`.omelette` and `.filling` parts are both *get* expressions.\n\nJust as we have two separate AST nodes for variable access and variable\nassignment, we need a second setter node to\ncomplement our getter node.\n\n^code set-ast (1 before, 1 after)\n\n\n\nIn case you don't remember, the way we handle assignment in the parser is a\nlittle funny. We can't easily tell that a series of tokens is the left-hand side\nof an assignment until we reach the `=`. Now that our assignment grammar rule\nhas `call` on the left side, which can expand to arbitrarily large expressions,\nthat final `=` may be many tokens away from the point where we need to know\nwe're parsing an assignment.\n\nInstead, the trick we do is parse the left-hand side as a normal expression.\nThen, when we stumble onto the equal sign after it, we take the expression we\nalready parsed and transform it into the correct syntax tree node for the\nassignment.\n\nWe add another clause to that transformation to handle turning an Expr.Get\nexpression on the left into the corresponding Expr.Set.\n\n^code assign-set (1 before, 1 after)\n\nThat's parsing our syntax. We push that node through into the resolver.\n\n^code resolver-visit-set\n\nAgain, like Expr.Get, the property itself is dynamically evaluated, so there's\nnothing to resolve there. All we need to do is recurse into the two\nsubexpressions of Expr.Set, the object whose property is being set, and the\nvalue it's being set to.\n\nThat leads us to the interpreter.\n\n^code interpreter-visit-set\n\nWe evaluate the object whose property is being set and check to see if it's a\nLoxInstance. If not, that's a runtime error. Otherwise, we evaluate the value\nbeing set and store it on the instance. That relies on a new method in\nLoxInstance.\n\n\n\n^code lox-instance-set-property\n\nNo real magic here. We stuff the values straight into the Java map where fields\nlive. Since Lox allows freely creating new fields on instances, there's no need\nto see if the key is already present.\n\n## Methods on Classes\n\nYou can create instances of classes and stuff data into them, but the class\nitself doesn't really *do* anything. Instances are just maps and all instances\nare more or less the same. To make them feel like instances *of classes*, we\nneed behavior -- methods.\n\nOur helpful parser already parses method declarations, so we're good there. We\nalso don't need to add any new parser support for method *calls*. We already\nhave `.` (getters) and `()` (function calls). A \"method call\" simply chains\nthose together.\n\n\"The\n\nThat raises an interesting question. What happens when those two expressions are\npulled apart? Assuming that `method` in this example is a method on the class of\n`object` and not a field on the instance, what should the following piece of\ncode do?\n\n```lox\nvar m = object.method;\nm(argument);\n```\n\nThis program \"looks up\" the method and stores the result -- whatever that is --\nin a variable and then calls that object later. Is this allowed? Can you treat a\nmethod like it's a function on the instance?\n\nWhat about the other direction?\n\n```lox\nclass Box {}\n\nfun notMethod(argument) {\n print \"called function with \" + argument;\n}\n\nvar box = Box();\nbox.function = notMethod;\nbox.function(\"argument\");\n```\n\nThis program creates an instance and then stores a function in a field on it.\nThen it calls that function using the same syntax as a method call. Does that\nwork?\n\nDifferent languages have different answers to these questions. One could write a\ntreatise on it. For Lox, we'll say the answer to both of these is yes, it does\nwork. We have a couple of reasons to justify that. For the second example --\ncalling a function stored in a field -- we want to support that because\nfirst-class functions are useful and storing them in fields is a perfectly\nnormal thing to do.\n\nThe first example is more obscure. One motivation is that users generally expect\nto be able to hoist a subexpression out into a local variable without changing\nthe meaning of the program. You can take this:\n\n```lox\nbreakfast(omelette.filledWith(cheese), sausage);\n```\n\nAnd turn it into this:\n\n```lox\nvar eggs = omelette.filledWith(cheese);\nbreakfast(eggs, sausage);\n```\n\nAnd it does the same thing. Likewise, since the `.` and the `()` in a method\ncall *are* two separate expressions, it seems you should be able to hoist the\n*lookup* part into a variable and then call it later. We need to think carefully about what the *thing*\nyou get when you look up a method is, and how it behaves, even in weird cases\nlike:\n\n\n\n```lox\nclass Person {\n sayName() {\n print this.name;\n }\n}\n\nvar jane = Person();\njane.name = \"Jane\";\n\nvar method = jane.sayName;\nmethod(); // ?\n```\n\nIf you grab a handle to a method on some instance and call it later, does it\n\"remember\" the instance it was pulled off from? Does `this` inside the method\nstill refer to that original object?\n\nHere's a more pathological example to bend your brain:\n\n```lox\nclass Person {\n sayName() {\n print this.name;\n }\n}\n\nvar jane = Person();\njane.name = \"Jane\";\n\nvar bill = Person();\nbill.name = \"Bill\";\n\nbill.sayName = jane.sayName;\nbill.sayName(); // ?\n```\n\nDoes that last line print \"Bill\" because that's the instance that we *called*\nthe method through, or \"Jane\" because it's the instance where we first grabbed\nthe method?\n\nEquivalent code in Lua and JavaScript would print \"Bill\". Those languages don't\nreally have a notion of \"methods\". Everything is sort of functions-in-fields, so\nit's not clear that `jane` \"owns\" `sayName` any more than `bill` does.\n\nLox, though, has real class syntax so we do know which callable things are\nmethods and which are functions. Thus, like Python, C#, and others, we will have\nmethods \"bind\" `this` to the original instance when the method is first grabbed.\nPython calls these **bound methods**.\n\n\n\nIn practice, that's usually what you want. If you take a reference to a method\non some object so you can use it as a callback later, you want to remember the\ninstance it belonged to, even if that callback happens to be stored in a field\non some other object.\n\nOK, that's a lot of semantics to load into your head. Forget about the edge\ncases for a bit. We'll get back to those. For now, let's get basic method calls\nworking. We're already parsing the method declarations inside the class body, so\nthe next step is to resolve them.\n\n^code resolve-methods (1 before, 1 after)\n\n\n\nWe iterate through the methods in the class body and call the\n`resolveFunction()` method we wrote for handling function declarations already.\nThe only difference is that we pass in a new FunctionType enum value.\n\n^code function-type-method (1 before, 1 after)\n\nThat's going to be important when we resolve `this` expressions. For now, don't\nworry about it. The interesting stuff is in the interpreter.\n\n^code interpret-methods (1 before, 1 after)\n\nWhen we interpret a class declaration statement, we turn the syntactic\nrepresentation of the class -- its AST node -- into its runtime representation.\nNow, we need to do that for the methods contained in the class as well. Each\nmethod declaration blossoms into a LoxFunction object.\n\nWe take all of those and wrap them up into a map, keyed by the method names.\nThat gets stored in LoxClass.\n\n^code lox-class-methods (1 before, 3 after)\n\nWhere an instance stores state, the class stores behavior. LoxInstance has its\nmap of fields, and LoxClass gets a map of methods. Even though methods are\nowned by the class, they are still accessed through instances of that class.\n\n^code lox-instance-get-method (5 before, 2 after)\n\nWhen looking up a property on an instance, if we don't find a matching field, we look for a method with that name\non the instance's class. If found, we return that. This is where the distinction\nbetween \"field\" and \"property\" becomes meaningful. When accessing a property,\nyou might get a field -- a bit of state stored on the instance -- or you could\nhit a method defined on the instance's class.\n\nThe method is looked up using this:\n\n\n\n^code lox-class-find-method\n\nYou can probably guess this method is going to get more interesting later. For\nnow, a simple map lookup on the class's method table is enough to get us\nstarted. Give it a try:\n\n\n\n```lox\nclass Bacon {\n eat() {\n print \"Crunch crunch crunch!\";\n }\n}\n\nBacon().eat(); // Prints \"Crunch crunch crunch!\".\n```\n\n\n\n## This\n\nWe can define both behavior and state on objects, but they aren't tied together\nyet. Inside a method, we have no way to access the fields of the \"current\"\nobject -- the instance that the method was called on -- nor can we call other\nmethods on that same object.\n\nTo get at that instance, it needs a name. Smalltalk,\nRuby, and Swift use \"self\". Simula, C++, Java, and others use \"this\". Python\nuses \"self\" by convention, but you can technically call it whatever you like.\n\n\n\nFor Lox, since we generally hew to Java-ish style, we'll go with \"this\". Inside\na method body, a `this` expression evaluates to the instance that the method was\ncalled on. Or, more specifically, since methods are accessed and then invoked as\ntwo steps, it will refer to the object that the method was *accessed* from.\n\nThat makes our job harder. Peep at:\n\n```lox\nclass Egotist {\n speak() {\n print this;\n }\n}\n\nvar method = Egotist().speak;\nmethod();\n```\n\nOn the second-to-last line, we grab a reference to the `speak()` method off an\ninstance of the class. That returns a function, and that function needs to\nremember the instance it was pulled off of so that *later*, on the last line, it\ncan still find it when the function is called.\n\nWe need to take `this` at the point that the method is accessed and attach it to\nthe function somehow so that it stays around as long as we need it to. Hmm... a\nway to store some extra data that hangs around a function, eh? That sounds an\nawful lot like a *closure*, doesn't it?\n\nIf we defined `this` as a sort of hidden variable in an environment that\nsurrounds the function returned when looking up a method, then uses of `this` in\nthe body would be able to find it later. LoxFunction already has the ability to\nhold on to a surrounding environment, so we have the machinery we need.\n\nLet's walk through an example to see how it works:\n\n```lox\nclass Cake {\n taste() {\n var adjective = \"delicious\";\n print \"The \" + this.flavor + \" cake is \" + adjective + \"!\";\n }\n}\n\nvar cake = Cake();\ncake.flavor = \"German chocolate\";\ncake.taste(); // Prints \"The German chocolate cake is delicious!\".\n```\n\nWhen we first evaluate the class definition, we create a LoxFunction for\n`taste()`. Its closure is the environment surrounding the class, in this case\nthe global one. So the LoxFunction we store in the class's method map looks\nlike so:\n\n\"The\n\nWhen we evaluate the `cake.taste` get expression, we create a new environment\nthat binds `this` to the object the method is accessed from (here, `cake`). Then\nwe make a *new* LoxFunction with the same code as the original one but using\nthat new environment as its closure.\n\n\"The\n\nThis is the LoxFunction that gets returned when evaluating the get expression\nfor the method name. When that function is later called by a `()` expression,\nwe create an environment for the method body as usual.\n\n\"Calling\n\nThe parent of the body environment is the environment we created earlier to bind\n`this` to the current object. Thus any use of `this` inside the body\nsuccessfully resolves to that instance.\n\nReusing our environment code for implementing `this` also takes care of\ninteresting cases where methods and functions interact, like:\n\n```lox\nclass Thing {\n getCallback() {\n fun localFunction() {\n print this;\n }\n\n return localFunction;\n }\n}\n\nvar callback = Thing().getCallback();\ncallback();\n```\n\nIn, say, JavaScript, it's common to return a callback from inside a method. That\ncallback may want to hang on to and retain access to the original object -- the\n`this` value -- that the method was associated with. Our existing support for\nclosures and environment chains should do all this correctly.\n\nLet's code it up. The first step is adding new\nsyntax for `this`.\n\n^code this-ast (1 before, 1 after)\n\n\n\nParsing is simple since it's a single token which our lexer already\nrecognizes as a reserved word.\n\n^code parse-this (2 before, 2 after)\n\nYou can start to see how `this` works like a variable when we get to the\nresolver.\n\n^code resolver-visit-this\n\nWe resolve it exactly like any other local variable using \"this\" as the name for\nthe \"variable\". Of course, that's not going to work right now, because \"this\"\n*isn't* declared in any scope. Let's fix that over in `visitClassStmt()`.\n\n^code resolver-begin-this-scope (2 before, 1 after)\n\nBefore we step in and start resolving the method bodies, we push a new scope and\ndefine \"this\" in it as if it were a variable. Then, when we're done, we discard\nthat surrounding scope.\n\n^code resolver-end-this-scope (2 before, 1 after)\n\nNow, whenever a `this` expression is encountered (at least inside a method) it\nwill resolve to a \"local variable\" defined in an implicit scope just outside of\nthe block for the method body.\n\nThe resolver has a new *scope* for `this`, so the interpreter needs to create a\ncorresponding *environment* for it. Remember, we always have to keep the\nresolver's scope chains and the interpreter's linked environments in sync with\neach other. At runtime, we create the environment after we find the method on\nthe instance. We replace the previous line of code that simply returned the\nmethod's LoxFunction with this:\n\n^code lox-instance-bind-method (1 before, 3 after)\n\nNote the new call to `bind()`. That looks like so:\n\n^code bind-instance\n\nThere isn't much to it. We create a new environment nestled inside the method's\noriginal closure. Sort of a closure-within-a-closure. When the method is called,\nthat will become the parent of the method body's environment.\n\nWe declare \"this\" as a variable in that environment and bind it to the given\ninstance, the instance that the method is being accessed from. *Et voilà*, the\nreturned LoxFunction now carries around its own little persistent world where\n\"this\" is bound to the object.\n\nThe remaining task is interpreting those `this` expressions. Similar to the\nresolver, it is the same as interpreting a variable expression.\n\n^code interpreter-visit-this\n\nGo ahead and give it a try using that cake example from earlier. With less than\ntwenty lines of code, our interpreter handles `this` inside methods even in all\nof the weird ways it can interact with nested classes, functions inside methods,\nhandles to methods, etc.\n\n### Invalid uses of this\n\nWait a minute. What happens if you try to use `this` *outside* of a method? What\nabout:\n\n```lox\nprint this;\n```\n\nOr:\n\n```lox\nfun notAMethod() {\n print this;\n}\n```\n\nThere is no instance for `this` to point to if you're not in a method. We could\ngive it some default value like `nil` or make it a runtime error, but the user\nhas clearly made a mistake. The sooner they find and fix that mistake, the\nhappier they'll be.\n\nOur resolution pass is a fine place to detect this error statically. It already\ndetects `return` statements outside of functions. We'll do something similar for\n`this`. In the vein of our existing FunctionType enum, we define a new ClassType\none.\n\n^code class-type (1 before, 1 after)\n\nYes, it could be a Boolean. When we get to inheritance, it will get a third\nvalue, hence the enum right now. We also add a corresponding field,\n`currentClass`. Its value tells us if we are currently inside a class\ndeclaration while traversing the syntax tree. It starts out `NONE` which means\nwe aren't in one.\n\nWhen we begin to resolve a class declaration, we change that.\n\n^code set-current-class (1 before, 1 after)\n\nAs with `currentFunction`, we store the previous value of the field in a local\nvariable. This lets us piggyback onto the JVM to keep a stack of `currentClass`\nvalues. That way we don't lose track of the previous value if one class nests\ninside another.\n\nOnce the methods have been resolved, we \"pop\" that stack by restoring the old\nvalue.\n\n^code restore-current-class (2 before, 1 after)\n\nWhen we resolve a `this` expression, the `currentClass` field gives us the bit\nof data we need to report an error if the expression doesn't occur nestled\ninside a method body.\n\n^code this-outside-of-class (1 before, 1 after)\n\nThat should help users use `this` correctly, and it saves us from having to\nhandle misuse at runtime in the interpreter.\n\n## Constructors and Initializers\n\nWe can do almost everything with classes now, and as we near the end of the\nchapter we find ourselves strangely focused on a beginning. Methods and fields\nlet us encapsulate state and behavior together so that an object always *stays*\nin a valid configuration. But how do we ensure a brand new object *starts* in a\ngood state?\n\nFor that, we need constructors. I find them one of the trickiest parts of a\nlanguage to design, and if you peer closely at most other languages, you'll see\ncracks around object construction where the seams of\nthe design don't quite fit together perfectly. Maybe there's something\nintrinsically messy about the moment of birth.\n\n\n\n\"Constructing\" an object is actually a pair of operations:\n\n1. The runtime *allocates* the memory required for\n a fresh instance. In most languages, this operation is at a fundamental\n level beneath what user code is able to access.\n\n \n\n2. Then, a user-provided chunk of code is called which *initializes* the\n unformed object.\n\n[placement new]: https://en.wikipedia.org/wiki/Placement_syntax\n\nThe latter is what we tend to think of when we hear \"constructor\", but the\nlanguage itself has usually done some groundwork for us before we get to that\npoint. In fact, our Lox interpreter already has that covered when it creates a\nnew LoxInstance object.\n\nWe'll do the remaining part -- user-defined initialization -- now. Languages\nhave a variety of notations for the chunk of code that sets up a new object for\na class. C++, Java, and C# use a method whose name matches the class name. Ruby\nand Python call it `init()`. The latter is nice and short, so we'll do that.\n\nIn LoxClass's implementation of LoxCallable, we add a few more lines.\n\n^code lox-class-call-initializer (2 before, 1 after)\n\nWhen a class is called, after the LoxInstance is created, we look for an \"init\"\nmethod. If we find one, we immediately bind and invoke it just like a normal\nmethod call. The argument list is forwarded along.\n\nThat argument list means we also need to tweak how a class declares its arity.\n\n^code lox-initializer-arity (1 before, 1 after)\n\nIf there is an initializer, that method's arity determines how many arguments\nyou must pass when you call the class itself. We don't *require* a class to\ndefine an initializer, though, as a convenience. If you don't have an\ninitializer, the arity is still zero.\n\nThat's basically it. Since we bind the `init()` method before we call it, it has\naccess to `this` inside its body. That, along with the arguments passed to the\nclass, are all you need to be able to set up the new instance however you\ndesire.\n\n### Invoking init() directly\n\nAs usual, exploring this new semantic territory rustles up a few weird\ncreatures. Consider:\n\n```lox\nclass Foo {\n init() {\n print this;\n }\n}\n\nvar foo = Foo();\nprint foo.init();\n```\n\nCan you \"re-initialize\" an object by directly calling its `init()` method? If\nyou do, what does it return? A reasonable answer\nwould be `nil` since that's what it appears the body returns.\n\nHowever -- and I generally dislike compromising to satisfy the\nimplementation -- it will make clox's implementation of constructors much\neasier if we say that `init()` methods always return `this`, even when\ndirectly called. In order to keep jlox compatible with that, we add a little\nspecial case code in LoxFunction.\n\n\n\n^code return-this (2 before, 1 after)\n\nIf the function is an initializer, we override the actual return value and\nforcibly return `this`. That relies on a new `isInitializer` field.\n\n^code is-initializer-field (2 before, 2 after)\n\nWe can't simply see if the name of the LoxFunction is \"init\" because the user\ncould have defined a *function* with that name. In that case, there *is* no\n`this` to return. To avoid *that* weird edge case, we'll directly store whether\nthe LoxFunction represents an initializer method. That means we need to go back\nand fix the few places where we create LoxFunctions.\n\n^code construct-function (1 before, 1 after)\n\nFor actual function declarations, `isInitializer` is always false. For methods,\nwe check the name.\n\n^code interpreter-method-initializer (1 before, 1 after)\n\nAnd then in `bind()` where we create the closure that binds `this` to a method,\nwe pass along the original method's value.\n\n^code lox-function-bind-with-initializer (1 before, 1 after)\n\n### Returning from init()\n\nWe aren't out of the woods yet. We've been assuming that a user-written\ninitializer doesn't explicitly return a value because most constructors don't.\nWhat should happen if a user tries:\n\n```lox\nclass Foo {\n init() {\n return \"something else\";\n }\n}\n```\n\nIt's definitely not going to do what they want, so we may as well make it a\nstatic error. Back in the resolver, we add another case to FunctionType.\n\n^code function-type-initializer (1 before, 1 after)\n\nWe use the visited method's name to determine if we're resolving an initializer\nor not.\n\n^code resolver-initializer-type (1 before, 1 after)\n\nWhen we later traverse into a `return` statement, we check that field and make\nit an error to return a value from inside an `init()` method.\n\n^code return-in-initializer (1 before, 1 after)\n\nWe're *still* not done. We statically disallow returning a *value* from an\ninitializer, but you can still use an empty early `return`.\n\n```lox\nclass Foo {\n init() {\n return;\n }\n}\n```\n\nThat is actually kind of useful sometimes, so we don't want to disallow it\nentirely. Instead, it should return `this` instead of `nil`. That's an easy fix\nover in LoxFunction.\n\n^code early-return-this (1 before, 1 after)\n\nIf we're in an initializer and execute a `return` statement, instead of\nreturning the value (which will always be `nil`), we again return `this`.\n\nPhew! That was a whole list of tasks but our reward is that our little\ninterpreter has grown an entire programming paradigm. Classes, methods, fields,\n`this`, and constructors. Our baby language is looking awfully grown-up.\n\n
\n\n## Challenges\n\n1. We have methods on instances, but there is no way to define \"static\" methods\n that can be called directly on the class object itself. Add support for\n them. Use a `class` keyword preceding the method to indicate a static method\n that hangs off the class object.\n\n ```lox\n class Math {\n class square(n) {\n return n * n;\n }\n }\n\n print Math.square(3); // Prints \"9\".\n ```\n\n You can solve this however you like, but the \"[metaclasses][]\" used by\n Smalltalk and Ruby are a particularly elegant approach. *Hint: Make LoxClass\n extend LoxInstance and go from there.*\n\n2. Most modern languages support \"getters\" and \"setters\" -- members on a class\n that look like field reads and writes but that actually execute user-defined\n code. Extend Lox to support getter methods. These are declared without a\n parameter list. The body of the getter is executed when a property with that\n name is accessed.\n\n ```lox\n class Circle {\n init(radius) {\n this.radius = radius;\n }\n\n area {\n return 3.141592653 * this.radius * this.radius;\n }\n }\n\n var circle = Circle(4);\n print circle.area; // Prints roughly \"50.2655\".\n ```\n\n3. Python and JavaScript allow you to freely access an object's fields from\n outside of its own methods. Ruby and Smalltalk encapsulate instance state.\n Only methods on the class can access the raw fields, and it is up to the\n class to decide which state is exposed. Most statically typed languages\n offer modifiers like `private` and `public` to control which parts of a\n class are externally accessible on a per-member basis.\n\n What are the trade-offs between these approaches and why might a language\n prefer one or the other?\n\n[metaclasses]: https://en.wikipedia.org/wiki/Metaclass\n\n
\n\n
\n\n## Design Note: Prototypes and Power\n\nIn this chapter, we introduced two new runtime entities, LoxClass and\nLoxInstance. The former is where behavior for objects lives, and the latter is\nfor state. What if you could define methods right on a single object, inside\nLoxInstance? In that case, we wouldn't need LoxClass at all. LoxInstance would\nbe a complete package for defining the behavior and state of an object.\n\nWe'd still want some way, without classes, to reuse behavior across multiple\ninstances. We could let a LoxInstance [*delegate*][delegate] directly to another\nLoxInstance to reuse its fields and methods, sort of like inheritance.\n\nUsers would model their program as a constellation of objects, some of which\ndelegate to each other to reflect commonality. Objects used as delegates\nrepresent \"canonical\" or \"prototypical\" objects that others refine. The result\nis a simpler runtime with only a single internal construct, LoxInstance.\n\nThat's where the name **[prototypes][proto]** comes from for this paradigm. It\nwas invented by David Ungar and Randall Smith in a language called [Self][].\nThey came up with it by starting with Smalltalk and following the above mental\nexercise to see how much they could pare it down.\n\nPrototypes were an academic curiosity for a long time, a fascinating one that\ngenerated interesting research but didn't make a dent in the larger world of\nprogramming. That is, until Brendan Eich crammed prototypes into JavaScript,\nwhich then promptly took over the world. Many (many) words have been written about prototypes in JavaScript.\nWhether that shows that prototypes are brilliant or confusing -- or both! -- is\nan open question.\n\n\n\nI won't get into whether or not I think prototypes are a good idea for a\nlanguage. I've made languages that are [prototypal][finch] and\n[class-based][wren], and my opinions of both are complex. What I want to discuss\nis the role of *simplicity* in a language.\n\nPrototypes are simpler than classes -- less code for the language implementer to\nwrite, and fewer concepts for the user to learn and understand. Does that make\nthem better? We language nerds have a tendency to fetishize minimalism.\nPersonally, I think simplicity is only part of the equation. What we really want\nto give the user is *power*, which I define as:\n\n```text\npower = breadth × ease ÷ complexity\n```\n\nNone of these are precise numeric measures. I'm using math as analogy here, not\nactual quantification.\n\n* **Breadth** is the range of different things the language lets you express.\n C has a lot of breadth -- it's been used for everything from operating\n systems to user applications to games. Domain-specific languages like\n AppleScript and Matlab have less breadth.\n\n* **Ease** is how little effort it takes to make the language do what you\n want. \"Usability\" might be another term, though it carries more baggage than\n I want to bring in. \"Higher-level\" languages tend to have more ease than\n \"lower-level\" ones. Most languages have a \"grain\" to them where some things\n feel easier to express than others.\n\n* **Complexity** is how big the language (including its runtime, core libraries,\n tools, ecosystem, etc.) is. People talk about how many pages are in a\n language's spec, or how many keywords it has. It's how much the user has to\n load into their wetware before they can be productive in the system. It is\n the antonym of simplicity.\n\n[proto]: https://en.wikipedia.org/wiki/Prototype-based_programming\n\nReducing complexity *does* increase power. The smaller the denominator, the\nlarger the resulting value, so our intuition that simplicity is good is valid.\nHowever, when reducing complexity, we must take care not to sacrifice breadth or\nease in the process, or the total power may go down. Java would be a strictly\n*simpler* language if it removed strings, but it probably wouldn't handle text\nmanipulation tasks well, nor would it be as easy to get things done.\n\nThe art, then, is finding *accidental* complexity that can be omitted --\nlanguage features and interactions that don't carry their weight by increasing\nthe breadth or ease of using the language.\n\nIf users want to express their program in terms of categories of objects, then\nbaking classes into the language increases the ease of doing that, hopefully by\na large enough margin to pay for the added complexity. But if that isn't how\nusers are using your language, then by all means leave classes out.\n\n
\n\n[delegate]: https://en.wikipedia.org/wiki/Prototype-based_programming#Delegation\n[prototypes]: http://gameprogrammingpatterns.com/prototype.html\n[self]: http://www.selflanguage.org/\n[finch]: http://finch.stuffwithstuff.com/\n[wren]: http://wren.io/\n" }, { "path": "book/closures.md", "content": "> As the man said, for every complex problem there's a simple solution, and it's\n> wrong.\n>\n> Umberto Eco, Foucault's Pendulum\n\nThanks to our diligent labor in [the last chapter][last], we have a virtual\nmachine with working functions. What it lacks is closures. Aside from global\nvariables, which are their own breed of animal, a function has no way to\nreference a variable declared outside of its own body.\n\n[last]: calls-and-functions.html\n\n```lox\nvar x = \"global\";\nfun outer() {\n var x = \"outer\";\n fun inner() {\n print x;\n }\n inner();\n}\nouter();\n```\n\nRun this example now and it prints \"global\". It's supposed to print \"outer\". To\nfix this, we need to include the entire lexical scope of all surrounding\nfunctions when resolving a variable.\n\nThis problem is harder in clox than it was in jlox because our bytecode VM\nstores locals on a stack. We used a stack because I claimed locals have stack\nsemantics -- variables are discarded in the reverse order that they are created.\nBut with closures, that's only *mostly* true.\n\n```lox\nfun makeClosure() {\n var local = \"local\";\n fun closure() {\n print local;\n }\n return closure;\n}\n\nvar closure = makeClosure();\nclosure();\n```\n\nThe outer function `makeClosure()` declares a variable, `local`. It also creates\nan inner function, `closure()` that captures that variable. Then `makeClosure()`\nreturns a reference to that function. Since the closure escapes while holding on to the local variable, `local` must\noutlive the function call where it was created.\n\n\n\nWe could solve this problem by dynamically allocating memory for all local\nvariables. That's what jlox does by putting everything in those Environment\nobjects that float around in Java's heap. But we don't want to. Using a stack is *really* fast. Most local variables are *not*\ncaptured by closures and do have stack semantics. It would suck to make all of\nthose slower for the benefit of the rare local that is captured.\n\n\n\nThis means a more complex approach than we used in our Java interpreter. Because\nsome locals have very different lifetimes, we will have two implementation\nstrategies. For locals that aren't used in closures, we'll keep them just as\nthey are on the stack. When a local is captured by a closure, we'll adopt\nanother solution that lifts them onto the heap where they can live as long as\nneeded.\n\nClosures have been around since the early Lisp days when bytes of memory and CPU\ncycles were more precious than emeralds. Over the intervening decades, hackers\ndevised all manner of ways to compile closures to\noptimized runtime representations. Some are more efficient but require a more\ncomplex compilation process than we could easily retrofit into clox.\n\n\n\nThe technique I explain here comes from the design of the Lua VM. It is fast,\nparsimonious with memory, and implemented with relatively little code. Even more\nimpressive, it fits naturally into the single-pass compilers clox and Lua both\nuse. It is somewhat intricate, though. It might take a while before all the\npieces click together in your mind. We'll build them one step at a time, and\nI'll try to introduce the concepts in stages.\n\n## Closure Objects\n\nOur VM represents functions at runtime using ObjFunction. These objects are\ncreated by the front end during compilation. At runtime, all the VM does is load\nthe function object from a constant table and bind it to a name. There is no\noperation to \"create\" a function at runtime. Much like string and number literals, they are constants instantiated purely at\ncompile time.\n\n\n\nThat made sense because all of the data that composes a function is known at\ncompile time: the chunk of bytecode compiled from the function's body, and the\nconstants used in the body. Once we introduce closures, though, that\nrepresentation is no longer sufficient. Take a gander at:\n\n```lox\nfun makeClosure(value) {\n fun closure() {\n print value;\n }\n return closure;\n}\n\nvar doughnut = makeClosure(\"doughnut\");\nvar bagel = makeClosure(\"bagel\");\ndoughnut();\nbagel();\n```\n\nThe `makeClosure()` function defines and returns a function. We call it twice\nand get two closures back. They are created by the same nested function\ndeclaration, `closure`, but close over different values. When we call the two\nclosures, each prints a different string. That implies we need some runtime\nrepresentation for a closure that captures the local variables surrounding the\nfunction as they exist when the function declaration is *executed*, not just\nwhen it is compiled.\n\nWe'll work our way up to capturing variables, but a good first step is defining\nthat object representation. Our existing ObjFunction type represents the \"raw\" compile-time state of a function declaration, since all\nclosures created from a single declaration share the same code and constants. At\nruntime, when we execute a function declaration, we wrap the ObjFunction in a\nnew ObjClosure structure. The latter has a reference to the underlying bare\nfunction along with runtime state for the variables the function closes over.\n\n\n\n\"An\n\nWe'll wrap every function in an ObjClosure, even if the function doesn't\nactually close over and capture any surrounding local variables. This is a\nlittle wasteful, but it simplifies the VM because we can always assume that the\nfunction we're calling is an ObjClosure. That new struct starts out like this:\n\n^code obj-closure\n\nRight now, it simply points to an ObjFunction and adds the necessary object\nheader stuff. Grinding through the usual ceremony for adding a new object type\nto clox, we declare a C function to create a new closure.\n\n^code new-closure-h (2 before, 1 after)\n\nThen we implement it here:\n\n^code new-closure\n\nIt takes a pointer to the ObjFunction it wraps. It also initializes the type\nfield to a new type.\n\n^code obj-type-closure (1 before, 1 after)\n\nAnd when we're done with a closure, we release its memory.\n\n^code free-closure (1 before, 1 after)\n\nWe free only the ObjClosure itself, not the ObjFunction. That's because the\nclosure doesn't *own* the function. There may be multiple closures that all\nreference the same function, and none of them claims any special privilege over\nit. We can't free the ObjFunction until *all* objects referencing it are gone --\nincluding even the surrounding function whose constant table contains it.\nTracking that sounds tricky, and it is! That's why we'll write a garbage\ncollector soon to manage it for us.\n\nWe also have the usual macros for checking a value's\ntype.\n\n\n\n^code is-closure (2 before, 1 after)\n\nAnd to cast a value:\n\n^code as-closure (2 before, 1 after)\n\nClosures are first-class objects, so you can print them.\n\n^code print-closure (1 before, 1 after)\n\nThey display exactly as ObjFunction does. From the user's perspective, the\ndifference between ObjFunction and ObjClosure is purely a hidden implementation\ndetail. With that out of the way, we have a working but empty representation for\nclosures.\n\n### Compiling to closure objects\n\nWe have closure objects, but our VM never creates them. The next step is getting\nthe compiler to emit instructions to tell the runtime when to create a new\nObjClosure to wrap a given ObjFunction. This happens right at the end of a\nfunction declaration.\n\n^code emit-closure (1 before, 1 after)\n\nBefore, the final bytecode for a function declaration was a single `OP_CONSTANT`\ninstruction to load the compiled function from the surrounding function's\nconstant table and push it onto the stack. Now we have a new instruction.\n\n^code closure-op (1 before, 1 after)\n\nLike `OP_CONSTANT`, it takes a single operand that represents a constant table\nindex for the function. But when we get over to the runtime implementation, we\ndo something more interesting.\n\nFirst, let's be diligent VM hackers and slot in disassembler support for the\ninstruction.\n\n^code disassemble-closure (2 before, 1 after)\n\nThere's more going on here than we usually have in the disassembler. By the end\nof the chapter, you'll discover that `OP_CLOSURE` is quite an unusual\ninstruction. It's straightforward right now -- just a single byte operand -- but\nwe'll be adding to it. This code here anticipates that future.\n\n### Interpreting function declarations\n\nMost of the work we need to do is in the runtime. We have to handle the new\ninstruction, naturally. But we also need to touch every piece of code in the VM\nthat works with ObjFunction and change it to use ObjClosure instead -- function\ncalls, call frames, etc. We'll start with the instruction, though.\n\n^code interpret-closure (1 before, 1 after)\n\nLike the `OP_CONSTANT` instruction we used before, first we load the compiled\nfunction from the constant table. The difference now is that we wrap that\nfunction in a new ObjClosure and push the result onto the stack.\n\nOnce you have a closure, you'll eventually want to call it.\n\n^code call-value-closure (1 before, 1 after)\n\nWe remove the code for calling objects whose type is `OBJ_FUNCTION`. Since we\nwrap all functions in ObjClosures, the runtime will never try to invoke a bare\nObjFunction anymore. Those objects live only in constant tables and get\nimmediately wrapped in closures before anything else\nsees them.\n\n\n\nWe replace the old code with very similar code for calling a closure instead.\nThe only difference is the type of object we pass to `call()`. The real changes\nare over in that function. First, we update its signature.\n\n^code call-signature (1 after)\n\nThen, in the body, we need to fix everything that referenced the function to\nhandle the fact that we've introduced a layer of indirection. We start with the\narity checking:\n\n^code check-arity (1 before, 1 after)\n\nThe only change is that we unwrap the closure to get to the underlying function.\nThe next thing `call()` does is create a new CallFrame. We change that code to\nstore the closure in the CallFrame and get the bytecode pointer from the\nclosure's function.\n\n^code call-init-closure (1 before, 1 after)\n\nThis necessitates changing the declaration of CallFrame too.\n\n^code call-frame-closure (1 before, 1 after)\n\nThat change triggers a few other cascading changes. Every place in the VM that\naccessed CallFrame's function needs to use a closure instead. First, the macro\nfor reading a constant from the current function's constant table:\n\n^code read-constant (2 before, 2 after)\n\nWhen `DEBUG_TRACE_EXECUTION` is enabled, it needs to get to the chunk from the\nclosure.\n\n^code disassemble-instruction (1 before, 1 after)\n\nLikewise when reporting a runtime error:\n\n^code runtime-error-function (1 before, 1 after)\n\nAlmost there. The last piece is the blob of code that sets up the very first\nCallFrame to begin executing the top-level code for a Lox script.\n\n^code interpret (1 before, 2 after)\n\nThe compiler still returns a raw ObjFunction when\ncompiling a script. That's fine, but it means we need to wrap it in an\nObjClosure here, before the VM can execute it.\n\n\n\nWe are back to a working interpreter. The *user* can't tell any difference, but\nthe compiler now generates code telling the VM to create a closure for each\nfunction declaration. Every time the VM executes a function declaration, it\nwraps the ObjFunction in a new ObjClosure. The rest of the VM now handles those\nObjClosures floating around. That's the boring stuff out of the way. Now we're\nready to make these closures actually *do* something.\n\n## Upvalues\n\nOur existing instructions for reading and writing local variables are limited to\na single function's stack window. Locals from a surrounding function are outside\nof the inner function's window. We're going to need some new instructions.\n\nThe easiest approach might be an instruction that takes a relative stack slot\noffset that can reach *before* the current function's window. That would work if\nclosed-over variables were always on the stack. But as we saw earlier, these\nvariables sometimes outlive the function where they are declared. That means\nthey won't always be on the stack.\n\nThe next easiest approach, then, would be to take any local variable that gets\nclosed over and have it always live on the heap. When the local variable\ndeclaration in the surrounding function is executed, the VM would allocate\nmemory for it dynamically. That way it could live as long as needed.\n\nThis would be a fine approach if clox didn't have a single-pass compiler. But\nthat restriction we chose in our implementation makes things harder. Take a look\nat this example:\n\n```lox\nfun outer() {\n var x = 1; // (1)\n x = 2; // (2)\n fun inner() { // (3)\n print x;\n }\n inner();\n}\n```\n\nHere, the compiler compiles the declaration of `x` at `(1)` and emits code for\nthe assignment at `(2)`. It does that before reaching the declaration of\n`inner()` at `(3)` and discovering that `x` is in fact closed over. We don't\nhave an easy way to go back and fix that already-emitted code to treat `x`\nspecially. Instead, we want a solution that allows a closed-over variable to\nlive on the stack exactly like a normal local variable *until the point that it\nis closed over*.\n\nFortunately, thanks to the Lua dev team, we have a solution. We use a level of\nindirection that they call an **upvalue**. An upvalue refers to a local variable\nin an enclosing function. Every closure maintains an array of upvalues, one for\neach surrounding local variable that the closure uses.\n\nThe upvalue points back into the stack to where the variable it captured lives.\nWhen the closure needs to access a closed-over variable, it goes through the\ncorresponding upvalue to reach it. When a function declaration is first executed\nand we create a closure for it, the VM creates the array of upvalues and wires\nthem up to \"capture\" the surrounding local variables that the closure needs.\n\nFor example, if we throw this program at clox,\n\n```lox\n{\n var a = 3;\n fun f() {\n print a;\n }\n}\n```\n\nthe compiler and runtime will conspire together to build up a set of objects in\nmemory like this:\n\n\"The\n\n\nThat might look overwhelming, but fear not. We'll work our way through it. The\nimportant part is that upvalues serve as the layer of indirection needed to\ncontinue to find a captured local variable even after it moves off the stack.\nBut before we get to all that, let's focus on compiling captured variables.\n\n### Compiling upvalues\n\nAs usual, we want to do as much work as possible during compilation to keep\nexecution simple and fast. Since local variables are lexically scoped in Lox, we\nhave enough knowledge at compile time to resolve which surrounding local\nvariables a function accesses and where those locals are declared. That, in\nturn, means we know *how many* upvalues a closure needs, *which* variables they\ncapture, and *which stack slots* contain those variables in the declaring\nfunction's stack window.\n\nCurrently, when the compiler resolves an identifier, it walks the block scopes\nfor the current function from innermost to outermost. If we don't find the\nvariable in that function, we assume the variable must be a global. We don't\nconsider the local scopes of enclosing functions -- they get skipped right over.\nThe first change, then, is inserting a resolution step for those outer local\nscopes.\n\n^code named-variable-upvalue (3 before, 1 after)\n\nThis new `resolveUpvalue()` function looks for a local variable declared in any\nof the surrounding functions. If it finds one, it returns an \"upvalue index\" for\nthat variable. (We'll get into what that means later.) Otherwise, it returns -1\nto indicate the variable wasn't found. If it was found, we use these two new\ninstructions for reading or writing to the variable through its upvalue:\n\n^code upvalue-ops (1 before, 1 after)\n\nWe're implementing this sort of top-down, so I'll show you how these work at\nruntime soon. The part to focus on now is how the compiler actually resolves the\nidentifier.\n\n^code resolve-upvalue\n\nWe call this after failing to resolve a local variable in the current function's\nscope, so we know the variable isn't in the current compiler. Recall that\nCompiler stores a pointer to the Compiler for the enclosing function, and these\npointers form a linked chain that goes all the way to the root Compiler for the\ntop-level code. Thus, if the enclosing Compiler is `NULL`, we know we've reached\nthe outermost function without finding a local variable. The variable must be\nglobal, so we return -1.\n\n\n\nOtherwise, we try to resolve the identifier as a *local* variable in the\n*enclosing* compiler. In other words, we look for it right outside the current\nfunction. For example:\n\n```lox\nfun outer() {\n var x = 1;\n fun inner() {\n print x; // (1)\n }\n inner();\n}\n```\n\nWhen compiling the identifier expression at `(1)`, `resolveUpvalue()` looks for\na local variable `x` declared in `outer()`. If found -- like it is in this\nexample -- then we've successfully resolved the variable. We create an upvalue\nso that the inner function can access the variable through that. The upvalue is\ncreated here:\n\n^code add-upvalue\n\nThe compiler keeps an array of upvalue structures to track the closed-over\nidentifiers that it has resolved in the body of each function. Remember how the\ncompiler's Local array mirrors the stack slot indexes where locals live at\nruntime? This new upvalue array works the same way. The indexes in the\ncompiler's array match the indexes where upvalues will live in the ObjClosure at\nruntime.\n\nThis function adds a new upvalue to that array. It also keeps track of the\nnumber of upvalues the function uses. It stores that count directly in the\nObjFunction itself because we'll also need that\nnumber for use at runtime.\n\n\n\nThe `index` field tracks the closed-over local variable's slot index. That way\nthe compiler knows *which* variable in the enclosing function needs to be\ncaptured. We'll circle back to what that `isLocal` field is for before too long.\nFinally, `addUpvalue()` returns the index of the created upvalue in the\nfunction's upvalue list. That index becomes the operand to the `OP_GET_UPVALUE`\nand `OP_SET_UPVALUE` instructions.\n\nThat's the basic idea for resolving upvalues, but the function isn't fully\nbaked. A closure may reference the same variable in a surrounding function\nmultiple times. In that case, we don't want to waste time and memory creating a\nseparate upvalue for each identifier expression. To fix that, before we add a\nnew upvalue, we first check to see if the function already has an upvalue that\ncloses over that variable.\n\n^code existing-upvalue (1 before, 1 after)\n\nIf we find an upvalue in the array whose slot index matches the one we're\nadding, we just return that *upvalue* index and reuse it. Otherwise, we fall\nthrough and add the new upvalue.\n\nThese two functions access and modify a bunch of new state, so let's define\nthat. First, we add the upvalue count to ObjFunction.\n\n^code upvalue-count (1 before, 1 after)\n\nWe're conscientious C programmers, so we zero-initialize that when an\nObjFunction is first allocated.\n\n^code init-upvalue-count (1 before, 1 after)\n\nIn the compiler, we add a field for the upvalue array.\n\n^code upvalues-array (1 before, 1 after)\n\nFor simplicity, I gave it a fixed size. The `OP_GET_UPVALUE` and\n`OP_SET_UPVALUE` instructions encode an upvalue index using a single byte\noperand, so there's a restriction on how many upvalues a function can have --\nhow many unique variables it can close over. Given that, we can afford a static\narray that large. We also need to make sure the compiler doesn't overflow that\nlimit.\n\n^code too-many-upvalues (5 before, 1 after)\n\nFinally, the Upvalue struct type itself.\n\n^code upvalue-struct\n\nThe `index` field stores which local slot the upvalue is capturing. The\n`isLocal` field deserves its own section, which we'll get to next.\n\n### Flattening upvalues\n\nIn the example I showed before, the closure is accessing a variable declared in\nthe immediately enclosing function. Lox also supports accessing local variables\ndeclared in *any* enclosing scope, as in:\n\n```lox\nfun outer() {\n var x = 1;\n fun middle() {\n fun inner() {\n print x;\n }\n }\n}\n```\n\nHere, we're accessing `x` in `inner()`. That variable is defined not in\n`middle()`, but all the way out in `outer()`. We need to handle cases like this\ntoo. You *might* think that this isn't much harder since the variable will\nsimply be somewhere farther down on the stack. But consider this devious example:\n\n\n\n```lox\nfun outer() {\n var x = \"value\";\n fun middle() {\n fun inner() {\n print x;\n }\n\n print \"create inner closure\";\n return inner;\n }\n\n print \"return from outer\";\n return middle;\n}\n\nvar mid = outer();\nvar in = mid();\nin();\n```\n\nWhen you run this, it should print:\n\n```text\nreturn from outer\ncreate inner closure\nvalue\n```\n\nI know, it's convoluted. The important part is that `outer()` -- where `x` is\ndeclared -- returns and pops all of its variables off the stack before the\n*declaration* of `inner()` executes. So, at the point in time that we create the\nclosure for `inner()`, `x` is already off the stack.\n\nHere, I traced out the execution flow for you:\n\n\"Tracing\n\nSee how `x` is popped ① before it is captured ② and then later\naccessed ③? We really have two problems:\n\n1. We need to resolve local variables that are declared in surrounding\n functions beyond the immediately enclosing one.\n\n2. We need to be able to capture variables that have already left the stack.\n\nFortunately, we're in the middle of adding upvalues to the VM, and upvalues are\nexplicitly designed for tracking variables that have escaped the stack. So, in a\nclever bit of self-reference, we can use upvalues to allow upvalues to capture\nvariables declared outside of the immediately surrounding function.\n\nThe solution is to allow a closure to capture either a local variable or *an\nexisting upvalue* in the immediately enclosing function. If a deeply nested\nfunction references a local variable declared several hops away, we'll thread it\nthrough all of the intermediate functions by having each function capture an\nupvalue for the next function to grab.\n\n\"An\n\nIn the above example, `middle()` captures the local variable `x` in the\nimmediately enclosing function `outer()` and stores it in its own upvalue. It\ndoes this even though `middle()` itself doesn't reference `x`. Then, when the\ndeclaration of `inner()` executes, its closure grabs the *upvalue* from the\nObjClosure for `middle()` that captured `x`. A function captures -- either a\nlocal or upvalue -- *only* from the immediately surrounding function, which is\nguaranteed to still be around at the point that the inner function declaration\nexecutes.\n\nIn order to implement this, `resolveUpvalue()` becomes recursive.\n\n^code resolve-upvalue-recurse (4 before, 1 after)\n\nIt's only another three lines of code, but I found this function really\nchallenging to get right the first time. This in spite of the fact that I wasn't\ninventing anything new, just porting the concept over from Lua. Most recursive\nfunctions either do all their work before the recursive call (a **pre-order\ntraversal**, or \"on the way down\"), or they do all the work after the recursive\ncall (a **post-order traversal**, or \"on the way back up\"). This function does\nboth. The recursive call is right in the middle.\n\nWe'll walk through it slowly. First, we look for a matching local variable in\nthe enclosing function. If we find one, we capture that local and return. That's\nthe base case.\n\n\n\nOtherwise, we look for a local variable beyond the immediately enclosing\nfunction. We do that by recursively calling `resolveUpvalue()` on the\n*enclosing* compiler, not the current one. This series of `resolveUpvalue()`\ncalls works its way along the chain of nested compilers until it hits one of\nthe base cases -- either it finds an actual local variable to capture or it\nruns out of compilers.\n\nWhen a local variable is found, the most deeply nested\ncall to `resolveUpvalue()` captures it and returns the upvalue index. That\nreturns to the next call for the inner function declaration. That call captures\nthe *upvalue* from the surrounding function, and so on. As each nested call to\n`resolveUpvalue()` returns, we drill back down into the innermost function\ndeclaration where the identifier we are resolving appears. At each step along\nthe way, we add an upvalue to the intervening function and pass the resulting\nupvalue index down to the next call.\n\n\n\nIt might help to walk through the original example when resolving `x`:\n\n\"Tracing\n\nNote that the new call to `addUpvalue()` passes `false` for the `isLocal`\nparameter. Now you see that that flag controls whether the closure captures a\nlocal variable or an upvalue from the surrounding function.\n\nBy the time the compiler reaches the end of a function declaration, every\nvariable reference has been resolved as either a local, an upvalue, or a global.\nEach upvalue may in turn capture a local variable from the surrounding function,\nor an upvalue in the case of transitive closures. We finally have enough data to\nemit bytecode which creates a closure at runtime that captures all of the\ncorrect variables.\n\n^code capture-upvalues (1 before, 1 after)\n\nThe `OP_CLOSURE` instruction is unique in that it has a variably sized encoding.\nFor each upvalue the closure captures, there are two single-byte operands. Each\npair of operands specifies what that upvalue captures. If the first byte is one,\nit captures a local variable in the enclosing function. If zero, it captures one\nof the function's upvalues. The next byte is the local slot or upvalue index to\ncapture.\n\nThis odd encoding means we need some bespoke support in the disassembly code\nfor `OP_CLOSURE`.\n\n^code disassemble-upvalues (1 before, 1 after)\n\nFor example, take this script:\n\n```lox\nfun outer() {\n var a = 1;\n var b = 2;\n fun middle() {\n var c = 3;\n var d = 4;\n fun inner() {\n print a + c + b + d;\n }\n }\n}\n```\n\nIf we disassemble the instruction that creates the closure for `inner()`, it\nprints this:\n\n```text\n0004 9 OP_CLOSURE 2 \n0006 | upvalue 0\n0008 | local 1\n0010 | upvalue 1\n0012 | local 2\n```\n\nWe have two other, simpler instructions to add disassembler support for.\n\n^code disassemble-upvalue-ops (2 before, 1 after)\n\nThese both have a single-byte operand, so there's nothing exciting going on. We\ndo need to add an include so the debug module can get to `AS_FUNCTION()`.\n\n^code debug-include-object (1 before, 1 after)\n\nWith that, our compiler is where we want it. For each function declaration, it\noutputs an `OP_CLOSURE` instruction followed by a series of operand byte pairs\nfor each upvalue it needs to capture at runtime. It's time to hop over to that\nside of the VM and get things running.\n\n## Upvalue Objects\n\nEach `OP_CLOSURE` instruction is now followed by the series of bytes that\nspecify the upvalues the ObjClosure should own. Before we process those\noperands, we need a runtime representation for upvalues.\n\n^code obj-upvalue\n\nWe know upvalues must manage closed-over variables that no longer live on the\nstack, which implies some amount of dynamic allocation. The easiest way to do\nthat in our VM is by building on the object system we already have. That way,\nwhen we implement a garbage collector in [the next chapter][gc], the GC can\nmanage memory for upvalues too.\n\n[gc]: garbage-collection.html\n\nThus, our runtime upvalue structure is an ObjUpvalue with the typical Obj header\nfield. Following that is a `location` field that points to the closed-over\nvariable. Note that this is a *pointer* to a Value, not a Value itself. It's a\nreference to a *variable*, not a *value*. This is important because it means\nthat when we assign to the variable the upvalue captures, we're assigning to the\nactual variable, not a copy. For example:\n\n```lox\nfun outer() {\n var x = \"before\";\n fun inner() {\n x = \"assigned\";\n }\n inner();\n print x;\n}\nouter();\n```\n\nThis program should print \"assigned\" even though the closure assigns to `x` and\nthe surrounding function accesses it.\n\nBecause upvalues are objects, we've got all the usual object machinery, starting\nwith a constructor-like function:\n\n^code new-upvalue-h (1 before, 1 after)\n\nIt takes the address of the slot where the closed-over variable lives. Here is\nthe implementation:\n\n^code new-upvalue\n\nWe simply initialize the object and store the pointer. That requires a new\nobject type.\n\n^code obj-type-upvalue (1 before, 1 after)\n\nAnd on the back side, a destructor-like function:\n\n^code free-upvalue (3 before, 1 after)\n\nMultiple closures can close over the same variable, so ObjUpvalue does not own\nthe variable it references. Thus, the only thing to free is the ObjUpvalue\nitself.\n\nAnd, finally, to print:\n\n^code print-upvalue (3 before, 1 after)\n\nPrinting isn't useful to end users. Upvalues are objects only so that we can\ntake advantage of the VM's memory management. They aren't first-class values\nthat a Lox user can directly access in a program. So this code will never\nactually execute... but it keeps the compiler from yelling at us about an\nunhandled switch case, so here we are.\n\n### Upvalues in closures\n\nWhen I first introduced upvalues, I said each closure has an array of them.\nWe've finally worked our way back to implementing that.\n\n^code upvalue-fields (1 before, 1 after)\n\nDifferent closures may have different numbers of\nupvalues, so we need a dynamic array. The upvalues themselves are dynamically\nallocated too, so we end up with a double pointer -- a pointer to a dynamically\nallocated array of pointers to upvalues. We also store the number of elements in\nthe array.\n\n\n\nWhen we create an ObjClosure, we allocate an upvalue array of the proper size,\nwhich we determined at compile time and stored in the ObjFunction.\n\n^code allocate-upvalue-array (1 before, 1 after)\n\nBefore creating the closure object itself, we allocate the array of upvalues and\ninitialize them all to `NULL`. This weird ceremony around memory is a careful\ndance to please the (forthcoming) garbage collection deities. It ensures the\nmemory manager never sees uninitialized memory.\n\nThen we store the array in the new closure, as well as copy the count over from\nthe ObjFunction.\n\n^code init-upvalue-fields (1 before, 1 after)\n\nWhen we free an ObjClosure, we also free the upvalue array.\n\n^code free-upvalues (1 before, 1 after)\n\nObjClosure does not own the ObjUpvalue objects themselves, but it does own *the\narray* containing pointers to those upvalues.\n\nWe fill the upvalue array over in the interpreter when it creates a closure.\nThis is where we walk through all of the operands after `OP_CLOSURE` to see what\nkind of upvalue each slot captures.\n\n^code interpret-capture-upvalues (1 before, 1 after)\n\nThis code is the magic moment when a closure comes to life. We iterate over each\nupvalue the closure expects. For each one, we read a pair of operand bytes. If\nthe upvalue closes over a local variable in the enclosing function, we let\n`captureUpvalue()` do the work.\n\nOtherwise, we capture an upvalue from the surrounding function. An `OP_CLOSURE`\ninstruction is emitted at the end of a function declaration. At the moment that\nwe are executing that declaration, the *current* function is the surrounding\none. That means the current function's closure is stored in the CallFrame at the\ntop of the callstack. So, to grab an upvalue from the enclosing function, we can\nread it right from the `frame` local variable, which caches a reference to that\nCallFrame.\n\nClosing over a local variable is more interesting. Most of the work happens in a\nseparate function, but first we calculate the argument to pass to it. We need to\ngrab a pointer to the captured local's slot in the surrounding function's stack\nwindow. That window begins at `frame->slots`, which points to slot zero. Adding\n`index` offsets that to the local slot we want to capture. We pass that pointer\nhere:\n\n^code capture-upvalue\n\nThis seems a little silly. All it does is create a new ObjUpvalue that captures\nthe given stack slot and returns it. Did we need a separate function for this?\nWell, no, not *yet*. But you know we are going to end up sticking more code in\nhere.\n\nFirst, let's wrap up what we're working on. Back in the interpreter code for\nhandling `OP_CLOSURE`, we eventually finish iterating through the upvalue\narray and initialize each one. When that completes, we have a new closure with\nan array full of upvalues pointing to variables.\n\nWith that in hand, we can implement the instructions that work with those\nupvalues.\n\n^code interpret-get-upvalue (1 before, 1 after)\n\nThe operand is the index into the current function's upvalue array. So we simply\nlook up the corresponding upvalue and dereference its location pointer to read\nthe value in that slot. Setting a variable is similar.\n\n^code interpret-set-upvalue (1 before, 1 after)\n\nWe take the value on top of the stack and store it\ninto the slot pointed to by the chosen upvalue. Just as with the instructions\nfor local variables, it's important that these instructions are fast. User\nprograms are constantly reading and writing variables, so if that's slow,\neverything is slow. And, as usual, the way we make them fast is by keeping them\nsimple. These two new instructions are pretty good: no control flow, no complex\narithmetic, just a couple of pointer indirections and a `push()`.\n\n\n\nThis is a milestone. As long as all of the variables remain on the stack, we\nhave working closures. Try this:\n\n```lox\nfun outer() {\n var x = \"outside\";\n fun inner() {\n print x;\n }\n inner();\n}\nouter();\n```\n\nRun this, and it correctly prints \"outside\".\n\n## Closed Upvalues\n\nOf course, a key feature of closures is that they hold on to the variable as\nlong as needed, even after the function that declares the variable has returned.\nHere's another example that *should* work:\n\n```lox\nfun outer() {\n var x = \"outside\";\n fun inner() {\n print x;\n }\n\n return inner;\n}\n\nvar closure = outer();\nclosure();\n```\n\nBut if you run it right now... who knows what it does? At runtime, it will end\nup reading from a stack slot that no longer contains the closed-over variable.\nLike I've mentioned a few times, the crux of the issue is that variables in\nclosures don't have stack semantics. That means we've got to hoist them off the\nstack when the function where they were declared returns. This final section of\nthe chapter does that.\n\n### Values and variables\n\nBefore we get to writing code, I want to dig into an important semantic point.\nDoes a closure close over a *value* or a *variable?* This isn't purely an academic question. I'm not just splitting hairs.\nConsider:\n\n\n\n```lox\nvar globalSet;\nvar globalGet;\n\nfun main() {\n var a = \"initial\";\n\n fun set() { a = \"updated\"; }\n fun get() { print a; }\n\n globalSet = set;\n globalGet = get;\n}\n\nmain();\nglobalSet();\nglobalGet();\n```\n\nThe outer `main()` function creates two closures and stores them in global variables so that they outlive the execution of\n`main()` itself. Both of those closures capture the same variable. The first\nclosure assigns a new value to it and the second closure reads the variable.\n\n\n\nWhat does the call to `globalGet()` print? If closures capture *values* then\neach closure gets its own copy of `a` with the value that `a` had at the point\nin time that the closure's function declaration executed. The call to\n`globalSet()` will modify `set()`'s copy of `a`, but `get()`'s copy will be\nunaffected. Thus, the call to `globalGet()` will print \"initial\".\n\nIf closures close over variables, then `get()` and `set()` will both capture --\nreference -- the *same mutable variable*. When `set()` changes `a`, it changes\nthe same `a` that `get()` reads from. There is only one `a`. That, in turn,\nimplies the call to `globalGet()` will print \"updated\".\n\nWhich is it? The answer for Lox and most other languages I know with closures is\nthe latter. Closures capture variables. You can think of them as capturing *the\nplace the value lives*. This is important to keep in mind as we deal with\nclosed-over variables that are no longer on the stack. When a variable moves to\nthe heap, we need to ensure that all closures capturing that variable retain a\nreference to its *one* new location. That way, when the variable is mutated, all\nclosures see the change.\n\n### Closing upvalues\n\nWe know that local variables always start out on the stack. This is faster, and\nlets our single-pass compiler emit code before it discovers the variable has\nbeen captured. We also know that closed-over variables need to move to the heap\nif the closure outlives the function where the captured variable is declared.\n\nFollowing Lua, we'll use **open upvalue** to refer to an upvalue that points to\na local variable still on the stack. When a variable moves to the heap, we are\n*closing* the upvalue and the result is, naturally, a **closed upvalue**. The\ntwo questions we need to answer are:\n\n1. Where on the heap does the closed-over variable go?\n\n2. When do we close the upvalue?\n\nThe answer to the first question is easy. We already have a convenient object on\nthe heap that represents a reference to a variable -- ObjUpvalue itself. The\nclosed-over variable will move into a new field right inside the ObjUpvalue\nstruct. That way we don't need to do any additional heap allocation to close an\nupvalue.\n\nThe second question is straightforward too. As long as the variable is on the\nstack, there may be code that refers to it there, and that code must work\ncorrectly. So the logical time to hoist the variable to the heap is as late as\npossible. If we move the local variable right when it goes out of scope, we are\ncertain that no code after that point will try to access it from the stack.\nAfter the variable is out of scope, the compiler will\nhave reported an error if any code tried to use it.\n\n\n\nThe compiler already emits an `OP_POP` instruction when a local variable goes\nout of scope. If a variable is captured by a closure, we will instead emit a\ndifferent instruction to hoist that variable out of the stack and into its\ncorresponding upvalue. To do that, the compiler needs to know which locals are closed over.\n\n\n\nThe compiler already maintains an array of Upvalue structs for each local\nvariable in the function to track exactly that state. That array is good for\nanswering \"Which variables does this closure use?\" But it's poorly suited for\nanswering, \"Does *any* function capture this local variable?\" In particular,\nonce the Compiler for some closure has finished, the Compiler for the enclosing\nfunction whose variable has been captured no longer has access to any of the\nupvalue state.\n\nIn other words, the compiler maintains pointers from upvalues to the locals they\ncapture, but not in the other direction. So we first need to add some extra\ntracking inside the existing Local struct so that we can tell if a given local\nis captured by a closure.\n\n^code is-captured-field (1 before, 1 after)\n\nThis field is `true` if the local is captured by any later nested function\ndeclaration. Initially, all locals are not captured.\n\n^code init-is-captured (1 before, 1 after)\n\nLikewise, the special \"slot zero local\" that the\ncompiler implicitly declares is not captured.\n\n\n\n^code init-zero-local-is-captured (1 before, 1 after)\n\nWhen resolving an identifier, if we end up creating an upvalue for a local\nvariable, we mark it as captured.\n\n^code mark-local-captured (1 before, 1 after)\n\nNow, at the end of a block scope when the compiler emits code to free the stack\nslots for the locals, we can tell which ones need to get hoisted onto the heap.\nWe'll use a new instruction for that.\n\n^code end-scope (3 before, 2 after)\n\nThe instruction requires no operand. We know that the variable will always be\nright on top of the stack at the point that this instruction executes. We\ndeclare the instruction.\n\n^code close-upvalue-op (1 before, 1 after)\n\nAnd add trivial disassembler support for it:\n\n^code disassemble-close-upvalue (1 before, 1 after)\n\nExcellent. Now the generated bytecode tells the runtime exactly when each\ncaptured local variable must move to the heap. Better, it does so only for the\nlocals that *are* used by a closure and need this special treatment. This aligns\nwith our general performance goal that we want users to pay only for\nfunctionality that they use. Variables that aren't used by closures live and die\nentirely on the stack just as they did before.\n\n### Tracking open upvalues\n\nLet's move over to the runtime side. Before we can interpret `OP_CLOSE_UPVALUE`\ninstructions, we have an issue to resolve. Earlier, when I talked about whether\nclosures capture variables or values, I said it was important that if multiple\nclosures access the same variable that they end up with a reference to the\nexact same storage location in memory. That way if one closure writes to the\nvariable, the other closure sees the change.\n\nRight now, if two closures capture the same local\nvariable, the VM creates a separate Upvalue for each one. The necessary sharing\nis missing. When we move the variable off the stack, if we move it into only one\nof the upvalues, the other upvalue will have an orphaned value.\n\n\n\nTo fix that, whenever the VM needs an upvalue that captures a particular local\nvariable slot, we will first search for an existing upvalue pointing to that\nslot. If found, we reuse that. The challenge is that all of the previously\ncreated upvalues are squirreled away inside the upvalue arrays of the various\nclosures. Those closures could be anywhere in the VM's memory.\n\nThe first step is to give the VM its own list of all open upvalues that point to\nvariables still on the stack. Searching a list each time the VM needs an upvalue\nsounds like it might be slow, but in practice, it's not bad. The number of\nvariables on the stack that actually get closed over tends to be small. And\nfunction declarations that create closures are rarely\non performance critical execution paths in the user's program.\n\n\n\nEven better, we can order the list of open upvalues by the stack slot index they\npoint to. The common case is that a slot has *not* already been captured --\nsharing variables between closures is uncommon -- and closures tend to capture\nlocals near the top of the stack. If we store the open upvalue array in stack\nslot order, as soon as we step past the slot where the local we're capturing\nlives, we know it won't be found. When that local is near the top of the stack,\nwe can exit the loop pretty early.\n\nMaintaining a sorted list requires inserting elements in the middle efficiently.\nThat suggests using a linked list instead of a dynamic array. Since we defined\nthe ObjUpvalue struct ourselves, the easiest implementation is an intrusive list\nthat puts the next pointer right inside the ObjUpvalue struct itself.\n\n^code next-field (1 before, 1 after)\n\nWhen we allocate an upvalue, it is not attached to any list yet so the link is\n`NULL`.\n\n^code init-next (1 before, 1 after)\n\nThe VM owns the list, so the head pointer goes right inside the main VM struct.\n\n^code open-upvalues-field (1 before, 1 after)\n\nThe list starts out empty.\n\n^code init-open-upvalues (1 before, 1 after)\n\nStarting with the first upvalue pointed to by the VM, each open upvalue points\nto the next open upvalue that references a local variable farther down the\nstack. This script, for example,\n\n```lox\n{\n var a = 1;\n fun f() {\n print a;\n }\n var b = 2;\n fun g() {\n print b;\n }\n var c = 3;\n fun h() {\n print c;\n }\n}\n```\n\nshould produce a series of linked upvalues like so:\n\n\"Three\n\nWhenever we close over a local variable, before creating a new upvalue, we look\nfor an existing one in the list.\n\n^code look-for-existing-upvalue (1 before, 1 after)\n\nWe start at the head of the list, which is the upvalue\nclosest to the top of the stack. We walk through the list, using a little\npointer comparison to iterate past every upvalue pointing to slots above the one\nwe're looking for. While we do that, we keep track of the preceding upvalue on\nthe list. We'll need to update that node's `next` pointer if we end up inserting\na node after it.\n\n\n\nThere are three reasons we can exit the loop:\n\n1. **The local slot we stopped at *is* the slot we're looking for.** We found\n an existing upvalue capturing the variable, so we reuse that upvalue.\n\n2. **We ran out of upvalues to search.** When `upvalue` is `NULL`, it means\n every open upvalue in the list points to locals above the slot we're looking\n for, or (more likely) the upvalue list is empty. Either way, we didn't find\n an upvalue for our slot.\n\n3. **We found an upvalue whose local slot is *below* the one we're looking\n for.** Since the list is sorted, that means we've gone past the slot we are\n closing over, and thus there must not be an existing upvalue for it.\n\nIn the first case, we're done and we've returned. Otherwise, we create a new\nupvalue for our local slot and insert it into the list at the right location.\n\n^code insert-upvalue-in-list (1 before, 1 after)\n\nThe current incarnation of this function already creates the upvalue, so we only\nneed to add code to insert the upvalue into the list. We exited the list\ntraversal by either going past the end of the list, or by stopping on the first\nupvalue whose stack slot is below the one we're looking for. In either case,\nthat means we need to insert the new upvalue *before* the object pointed at by\n`upvalue` (which may be `NULL` if we hit the end of the list).\n\nAs you may have learned in Data Structures 101, to insert a node into a linked\nlist, you set the `next` pointer of the previous node to point to your new one.\nWe have been conveniently keeping track of that preceding node as we walked the\nlist. We also need to handle the special case where\nwe are inserting a new upvalue at the head of the list, in which case the \"next\"\npointer is the VM's head pointer.\n\n\n\nWith this updated function, the VM now ensures that there is only ever a single\nObjUpvalue for any given local slot. If two closures capture the same variable,\nthey will get the same upvalue. We're ready to move those upvalues off the\nstack now.\n\n### Closing upvalues at runtime\n\nThe compiler helpfully emits an `OP_CLOSE_UPVALUE` instruction to tell the VM\nexactly when a local variable should be hoisted onto the heap. Executing that\ninstruction is the interpreter's responsibility.\n\n^code interpret-close-upvalue (1 before, 1 after)\n\nWhen we reach the instruction, the variable we are hoisting is right on top of\nthe stack. We call a helper function, passing the address of that stack slot.\nThat function is responsible for closing the upvalue and moving the local from\nthe stack to the heap. After that, the VM is free to discard the stack slot,\nwhich it does by calling `pop()`.\n\nThe fun stuff happens here:\n\n^code close-upvalues\n\nThis function takes a pointer to a stack slot. It closes every open upvalue it\ncan find that points to that slot or any slot above it on the stack. Right now,\nwe pass a pointer only to the top slot on the stack, so the \"or above it\" part\ndoesn't come into play, but it will soon.\n\nTo do this, we walk the VM's list of open upvalues, again from top to bottom. If\nan upvalue's location points into the range of slots we're closing, we close the\nupvalue. Otherwise, once we reach an upvalue outside of the range, we know the\nrest will be too, so we stop iterating.\n\nThe way an upvalue gets closed is pretty cool. First,\nwe copy the variable's value into the `closed` field in the ObjUpvalue. That's\nwhere closed-over variables live on the heap. The `OP_GET_UPVALUE` and\n`OP_SET_UPVALUE` instructions need to look for the variable there after it's\nbeen moved. We could add some conditional logic in the interpreter code for\nthose instructions to check some flag for whether the upvalue is open or closed.\n\nBut there is already a level of indirection in play -- those instructions\ndereference the `location` pointer to get to the variable's value. When the\nvariable moves from the stack to the `closed` field, we simply update that\n`location` to the address of the ObjUpvalue's *own* `closed` field.\n\n\n\n\"Moving\n\nWe don't need to change how `OP_GET_UPVALUE` and `OP_SET_UPVALUE` are\ninterpreted at all. That keeps them simple, which in turn keeps them fast. We do\nneed to add the new field to ObjUpvalue, though.\n\n^code closed-field (1 before, 1 after)\n\nAnd we should zero it out when we create an ObjUpvalue so there's no\nuninitialized memory floating around.\n\n^code init-closed (1 before, 1 after)\n\nWhenever the compiler reaches the end of a block, it discards all local\nvariables in that block and emits an `OP_CLOSE_UPVALUE` for each local variable\nthat was closed over. The compiler does *not* emit any\ninstructions at the end of the outermost block scope that defines a function\nbody. That scope contains the function's parameters and any locals declared\nimmediately inside the function. Those need to get closed too.\n\n\n\nThis is the reason `closeUpvalues()` accepts a pointer to a stack slot. When a\nfunction returns, we call that same helper and pass in the first stack slot\nowned by the function.\n\n^code return-close-upvalues (1 before, 1 after)\n\nBy passing the first slot in the function's stack window, we close every\nremaining open upvalue owned by the returning function. And with that, we now\nhave a fully functioning closure implementation. Closed-over variables live as\nlong as they are needed by the functions that capture them.\n\nThis was a lot of work! In jlox, closures fell out naturally from our\nenvironment representation. In clox, we had to add a lot of code -- new bytecode\ninstructions, more data structures in the compiler, and new runtime objects. The\nVM very much treats variables in closures as different from other variables.\n\nThere is a rationale for that. In terms of implementation complexity, jlox gave\nus closures \"for free\". But in terms of *performance*, jlox's closures are\nanything but. By allocating *all* environments on the heap, jlox pays a\nsignificant performance price for *all* local variables, even the majority which\nare never captured by closures.\n\nWith clox, we have a more complex system, but that allows us to tailor the\nimplementation to fit the two use patterns we observe for local variables. For\nmost variables which do have stack semantics, we allocate them entirely on the\nstack which is simple and fast. Then, for the few local variables where that\ndoesn't work, we have a second slower path we can opt in to as needed.\n\nFortunately, users don't perceive the complexity. From their perspective, local\nvariables in Lox are simple and uniform. The *language itself* is as simple as\njlox's implementation. But under the hood, clox is watching what the user does\nand optimizing for their specific uses. As your language implementations grow in\nsophistication, you'll find yourself doing this more. A large fraction of\n\"optimization\" is about adding special case code that detects certain uses and\nprovides a custom-built, faster path for code that fits that pattern.\n\nWe have lexical scoping fully working in clox now, which is a major milestone.\nAnd, now that we have functions and variables with complex lifetimes, we also\nhave a *lot* of objects floating around in clox's heap, with a web of pointers\nstringing them together. The [next step][] is figuring out how to manage that\nmemory so that we can free some of those objects when they're no longer needed.\n\n[next step]: garbage-collection.html\n\n
\n\n## Challenges\n\n1. Wrapping every ObjFunction in an ObjClosure introduces a level of\n indirection that has a performance cost. That cost isn't necessary for\n functions that do not close over any variables, but it does let the runtime\n treat all calls uniformly.\n\n Change clox to only wrap functions in ObjClosures that need upvalues. How\n does the code complexity and performance compare to always wrapping\n functions? Take care to benchmark programs that do and do not use closures.\n How should you weight the importance of each benchmark? If one gets slower\n and one faster, how do you decide what trade-off to make to choose an\n implementation strategy?\n\n2. Read the design note below. I'll wait. Now, how do you think Lox *should*\n behave? Change the implementation to create a new variable for each loop\n iteration.\n\n3. A [famous koan][koan] teaches us that \"objects are a poor man's closure\"\n (and vice versa). Our VM doesn't support objects yet, but now that we have\n closures we can approximate them. Using closures, write a Lox program that\n models two-dimensional vector \"objects\". It should:\n\n * Define a \"constructor\" function to create a new vector with the given\n *x* and *y* coordinates.\n\n * Provide \"methods\" to access the *x* and *y* coordinates of values\n returned from that constructor.\n\n * Define an addition \"method\" that adds two vectors and produces a third.\n\n\n[koan]: http://wiki.c2.com/?ClosuresAndObjectsAreEquivalent\n\n
\n\n
\n\n## Design Note: Closing Over the Loop Variable\n\nClosures capture variables. When two closures capture the same variable, they\nshare a reference to the same underlying storage location. This fact is visible\nwhen new values are assigned to the variable. Obviously, if two closures capture\n*different* variables, there is no sharing.\n\n```lox\nvar globalOne;\nvar globalTwo;\n\nfun main() {\n {\n var a = \"one\";\n fun one() {\n print a;\n }\n globalOne = one;\n }\n\n {\n var a = \"two\";\n fun two() {\n print a;\n }\n globalTwo = two;\n }\n}\n\nmain();\nglobalOne();\nglobalTwo();\n```\n\nThis prints \"one\" then \"two\". In this example, it's pretty clear that the two\n`a` variables are different. But it's not always so obvious. Consider:\n\n```lox\nvar globalOne;\nvar globalTwo;\n\nfun main() {\n for (var a = 1; a <= 2; a = a + 1) {\n fun closure() {\n print a;\n }\n if (globalOne == nil) {\n globalOne = closure;\n } else {\n globalTwo = closure;\n }\n }\n}\n\nmain();\nglobalOne();\nglobalTwo();\n```\n\nThe code is convoluted because Lox has no collection types. The important part\nis that the `main()` function does two iterations of a `for` loop. Each time\nthrough the loop, it creates a closure that captures the loop variable. It\nstores the first closure in `globalOne` and the second in `globalTwo`.\n\nThere are definitely two different closures. Do they close over two different\nvariables? Is there only one `a` for the entire duration of the loop, or does\neach iteration get its own distinct `a` variable?\n\nThe script here is strange and contrived, but this does show up in real code\nin languages that aren't as minimal as clox. Here's a JavaScript example:\n\n```js\nvar closures = [];\nfor (var i = 1; i <= 2; i++) {\n closures.push(function () { console.log(i); });\n}\n\nclosures[0]();\nclosures[1]();\n```\n\nDoes this print \"1\" then \"2\", or does it print \"3\"\ntwice? You may be surprised to hear that it prints \"3\" twice. In this JavaScript\nprogram, there is only a single `i` variable whose lifetime includes all\niterations of the loop, including the final exit.\n\n\n\nIf you're familiar with JavaScript, you probably know that variables declared\nusing `var` are implicitly *hoisted* to the surrounding function or top-level\nscope. It's as if you really wrote this:\n\n```js\nvar closures = [];\nvar i;\nfor (i = 1; i <= 2; i++) {\n closures.push(function () { console.log(i); });\n}\n\nclosures[0]();\nclosures[1]();\n```\n\nAt that point, it's clearer that there is only a single `i`. Now consider if\nyou change the program to use the newer `let` keyword:\n\n```js\nvar closures = [];\nfor (let i = 1; i <= 2; i++) {\n closures.push(function () { console.log(i); });\n}\n\nclosures[0]();\nclosures[1]();\n```\n\nDoes this new program behave the same? Nope. In this case, it prints \"1\" then\n\"2\". Each closure gets its own `i`. That's sort of strange when you think about\nit. The increment clause is `i++`. That looks very much like it is assigning to\nand mutating an existing variable, not creating a new one.\n\nLet's try some other languages. Here's Python:\n\n```python\nclosures = []\nfor i in range(1, 3):\n closures.append(lambda: print(i))\n\nclosures[0]()\nclosures[1]()\n```\n\nPython doesn't really have block scope. Variables are implicitly declared and\nare automatically scoped to the surrounding function. Kind of like hoisting in\nJS, now that I think about it. So both closures capture the same variable.\nUnlike C, though, we don't exit the loop by incrementing `i` *past* the last\nvalue, so this prints \"2\" twice.\n\nWhat about Ruby? Ruby has two typical ways to iterate numerically. Here's the\nclassic imperative style:\n\n```ruby\nclosures = []\nfor i in 1..2 do\n closures << lambda { puts i }\nend\n\nclosures[0].call\nclosures[1].call\n```\n\nThis, like Python, prints \"2\" twice. But the more idiomatic Ruby style is using\na higher-order `each()` method on range objects:\n\n```ruby\nclosures = []\n(1..2).each do |i|\n closures << lambda { puts i }\nend\n\nclosures[0].call\nclosures[1].call\n```\n\nIf you're not familiar with Ruby, the `do |i| ... end` part is basically a\nclosure that gets created and passed to the `each()` method. The `|i|` is the\nparameter signature for the closure. The `each()` method invokes that closure\ntwice, passing in 1 for `i` the first time and 2 the second time.\n\nIn this case, the \"loop variable\" is really a function parameter. And, since\neach iteration of the loop is a separate invocation of the function, those are\ndefinitely separate variables for each call. So this prints \"1\" then \"2\".\n\nIf a language has a higher-level iterator-based looping structure like `foreach`\nin C#, Java's \"enhanced for\", `for-of` in JavaScript, `for-in` in Dart, etc.,\nthen I think it's natural to the reader to have each iteration create a new\nvariable. The code *looks* like a new variable because the loop header looks\nlike a variable declaration. And there's no increment expression that looks like\nit's mutating that variable to advance to the next step.\n\nIf you dig around StackOverflow and other places, you find evidence that this is\nwhat users expect, because they are very surprised when they *don't* get it. In\nparticular, C# originally did *not* create a new loop variable for each\niteration of a `foreach` loop. This was such a frequent source of user confusion\nthat they took the very rare step of shipping a breaking change to the language.\nIn C# 5, each iteration creates a fresh variable.\n\nOld C-style `for` loops are harder. The increment clause really does look like\nmutation. That implies there is a single variable that's getting updated each\nstep. But it's almost never *useful* for each iteration to share a loop\nvariable. The only time you can even detect this is when closures capture it.\nAnd it's rarely helpful to have a closure that references a variable whose value\nis whatever value caused you to exit the loop.\n\nThe pragmatically useful answer is probably to do what JavaScript does with\n`let` in `for` loops. Make it look like mutation but actually create a new\nvariable each time, because that's what users want. It is kind of weird when you\nthink about it, though.\n\n
\n" }, { "path": "book/compiling-expressions.md", "content": "> In the middle of the journey of our life I found myself within a dark woods\n> where the straight way was lost.\n>\n> Dante Alighieri, Inferno\n\nThis chapter is exciting for not one, not two, but *three* reasons. First, it\nprovides the final segment of our VM's execution pipeline. Once in place, we can\nplumb the user's source code from scanning all the way through to executing it.\n\n\"Lowering\n\nSecond, we get to write an actual, honest-to-God *compiler*. It parses source\ncode and outputs a low-level series of binary instructions. Sure, it's bytecode and not some chip's native instruction set, but\nit's way closer to the metal than jlox was. We're about to be real language\nhackers.\n\n\n\nThird and finally, I get to show you one of my\nabsolute favorite algorithms: Vaughan Pratt's \"top-down operator precedence\nparsing\". It's the most elegant way I know to parse expressions. It gracefully\nhandles prefix operators, postfix, infix, *mixfix*, any kind of *-fix* you got.\nIt deals with precedence and associativity without breaking a sweat. I love it.\n\n\n\nAs usual, before we get to the fun stuff, we've got some preliminaries to work\nthrough. You have to eat your vegetables before you get dessert. First, let's\nditch that temporary scaffolding we wrote for testing the scanner and replace it\nwith something more useful.\n\n^code interpret-chunk (1 before, 1 after)\n\nWe create a new empty chunk and pass it over to the compiler. The compiler will\ntake the user's program and fill up the chunk with bytecode. At least, that's\nwhat it will do if the program doesn't have any compile errors. If it does\nencounter an error, `compile()` returns `false` and we discard the unusable\nchunk.\n\nOtherwise, we send the completed chunk over to the VM to be executed. When the\nVM finishes, we free the chunk and we're done. As you can see, the signature to\n`compile()` is different now.\n\n^code compile-h (2 before, 2 after)\n\nWe pass in the chunk where the compiler will write the code, and then\n`compile()` returns whether or not compilation succeeded. We make the same\nchange to the signature in the implementation.\n\n^code compile-signature (2 before, 1 after)\n\nThat call to `initScanner()` is the only line that survives this chapter. Rip\nout the temporary code we wrote to test the scanner and replace it with these\nthree lines:\n\n^code compile-chunk (1 before, 1 after)\n\nThe call to `advance()` \"primes the pump\" on the scanner. We'll see what it does\nsoon. Then we parse a single expression. We aren't going to do statements yet,\nso that's the only subset of the grammar we support. We'll revisit this when we\n[add statements in a few chapters][globals]. After we compile the expression, we\nshould be at the end of the source code, so we check for the sentinel EOF token.\n\n[globals]: global-variables.html\n\nWe're going to spend the rest of the chapter making this function work,\nespecially that little `expression()` call. Normally, we'd dive right into that\nfunction definition and work our way through the implementation from top to\nbottom.\n\nThis chapter is different. Pratt's parsing technique is\nremarkably simple once you have it all loaded in your head, but it's a little\ntricky to break into bite-sized pieces. It's recursive, of course, which is part\nof the problem. But it also relies on a big table of data. As we build up the\nalgorithm, that table grows additional columns.\n\n\n\nI don't want to revisit 40-something lines of code each time we extend the\ntable. So we're going to work our way into the core of the parser from the\noutside and cover all of the surrounding bits before we get to the juicy center.\nThis will require a little more patience and mental scratch space than most\nchapters, but it's the best I could do.\n\n## Single-Pass Compilation\n\nA compiler has roughly two jobs. It parses the user's source code to understand\nwhat it means. Then it takes that knowledge and outputs low-level instructions\nthat produce the same semantics. Many languages split those two roles into two\nseparate passes in the implementation. A parser\nproduces an AST -- just like jlox does -- and then a code generator traverses\nthe AST and outputs target code.\n\n\n\nIn clox, we're taking an old-school approach and merging these two passes into\none. Back in the day, language hackers did this because computers literally\ndidn't have enough memory to store an entire source file's AST. We're doing it\nbecause it keeps our compiler simpler, which is a real asset when programming in\nC.\n\nSingle-pass compilers like we're going to build don't work well for all\nlanguages. Since the compiler has only a peephole view into the user's program\nwhile generating code, the language must be designed such that you don't need\nmuch surrounding context to understand a piece of syntax. Fortunately, tiny,\ndynamically typed Lox is well-suited to that.\n\n\n\nWhat this means in practical terms is that our \"compiler\" C module has\nfunctionality you'll recognize from jlox for parsing -- consuming tokens,\nmatching expected token types, etc. And it also has functions for code gen --\nemitting bytecode and adding constants to the destination chunk. (And it means\nI'll use \"parsing\" and \"compiling\" interchangeably throughout this and later\nchapters.)\n\nWe'll build the parsing and code generation halves first. Then we'll stitch them\ntogether with the code in the middle that uses Pratt's technique to parse Lox's\nparticular grammar and output the right bytecode.\n\n## Parsing Tokens\n\nFirst up, the front half of the compiler. This function's name should sound\nfamiliar.\n\n^code advance (1 before)\n\nJust like in jlox, it steps forward through the token stream. It asks the\nscanner for the next token and stores it for later use. Before doing that, it\ntakes the old `current` token and stashes that in a `previous` field. That will\ncome in handy later so that we can get at the lexeme after we match a token.\n\nThe code to read the next token is wrapped in a loop. Remember, clox's scanner\ndoesn't report lexical errors. Instead, it creates special *error tokens* and\nleaves it up to the parser to report them. We do that here.\n\nWe keep looping, reading tokens and reporting the errors, until we hit a\nnon-error one or reach the end. That way, the rest of the parser sees only valid\ntokens. The current and previous token are stored in this struct:\n\n^code parser (1 before, 2 after)\n\nLike we did in other modules, we have a single global variable of this struct\ntype so we don't need to pass the state around from function to function in the\ncompiler.\n\n### Handling syntax errors\n\nIf the scanner hands us an error token, we need to actually tell the user. That\nhappens using this:\n\n^code error-at-current\n\nWe pull the location out of the current token in order to tell the user where\nthe error occurred and forward it to `errorAt()`. More often, we'll report an\nerror at the location of the token we just consumed, so we give the shorter name\nto this other function:\n\n^code error\n\nThe actual work happens here:\n\n^code error-at\n\nFirst, we print where the error occurred. We try to show the lexeme if it's\nhuman-readable. Then we print the error message itself. After that, we set this\n`hadError` flag. That records whether any errors occurred during compilation.\nThis field also lives in the parser struct.\n\n^code had-error-field (1 before, 1 after)\n\nEarlier I said that `compile()` should return `false` if an error occurred. Now\nwe can make it do that.\n\n^code return-had-error (1 before, 1 after)\n\nI've got another flag to introduce for error handling. We want to avoid error\ncascades. If the user has a mistake in their code and the parser gets confused\nabout where it is in the grammar, we don't want it to spew out a whole pile of\nmeaningless knock-on errors after the first one.\n\nWe fixed that in jlox using panic mode error recovery. In the Java interpreter,\nwe threw an exception to unwind out of all of the parser code to a point where\nwe could skip tokens and resynchronize. We don't have exceptions in C. Instead, we'll do a little smoke and\nmirrors. We add a flag to track whether we're currently in panic mode.\n\n\n\n^code panic-mode-field (1 before, 1 after)\n\nWhen an error occurs, we set it.\n\n^code set-panic-mode (1 before, 1 after)\n\nAfter that, we go ahead and keep compiling as normal as if the error never\noccurred. The bytecode will never get executed, so it's harmless to keep on\ntrucking. The trick is that while the panic mode flag is set, we simply suppress\nany other errors that get detected.\n\n^code check-panic-mode (1 before, 1 after)\n\nThere's a good chance the parser will go off in the weeds, but the user won't\nknow because the errors all get swallowed. Panic mode ends when the parser\nreaches a synchronization point. For Lox, we chose statement boundaries, so when\nwe later add those to our compiler, we'll clear the flag there.\n\nThese new fields need to be initialized.\n\n^code init-parser-error (1 before, 1 after)\n\nAnd to display the errors, we need a standard header.\n\n^code compiler-include-stdlib (1 before, 2 after)\n\nThere's one last parsing function, another old friend from jlox.\n\n^code consume\n\nIt's similar to `advance()` in that it reads the next token. But it also\nvalidates that the token has an expected type. If not, it reports an error. This\nfunction is the foundation of most syntax errors in the compiler.\n\nOK, that's enough on the front end for now.\n\n## Emitting Bytecode\n\nAfter we parse and understand a piece of the user's program, the next step is to\ntranslate that to a series of bytecode instructions. It starts with the easiest\npossible step: appending a single byte to the chunk.\n\n^code emit-byte\n\nIt's hard to believe great things will flow through such a simple function. It\nwrites the given byte, which may be an opcode or an operand to an instruction.\nIt sends in the previous token's line information so that runtime errors are\nassociated with that line.\n\nThe chunk that we're writing gets passed into `compile()`, but it needs to make\nits way to `emitByte()`. To do that, we rely on this intermediary function:\n\n^code compiling-chunk (1 before, 1 after)\n\nRight now, the chunk pointer is stored in a module-level variable like we store\nother global state. Later, when we start compiling user-defined functions, the\nnotion of \"current chunk\" gets more complicated. To avoid having to go back and\nchange a lot of code, I encapsulate that logic in the `currentChunk()` function.\n\nWe initialize this new module variable before we write any bytecode:\n\n^code init-compile-chunk (2 before, 2 after)\n\nThen, at the very end, when we're done compiling the chunk, we wrap things up.\n\n^code finish-compile (1 before, 1 after)\n\nThat calls this:\n\n^code end-compiler\n\nIn this chapter, our VM deals only with expressions. When you run clox, it will\nparse, compile, and execute a single expression, then print the result. To print\nthat value, we are temporarily using the `OP_RETURN` instruction. So we have the\ncompiler add one of those to the end of the chunk.\n\n^code emit-return\n\nWhile we're here in the back end we may as well make our lives easier.\n\n^code emit-bytes\n\nOver time, we'll have enough cases where we need to write an opcode followed by\na one-byte operand that it's worth defining this convenience function.\n\n## Parsing Prefix Expressions\n\nWe've assembled our parsing and code generation utility functions. The missing\npiece is the code in the middle that connects those together.\n\n\"Parsing\n\nThe only step in `compile()` that we have left to implement is this function:\n\n^code expression\n\nWe aren't ready to implement every kind of expression in Lox yet. Heck, we don't\neven have Booleans. For this chapter, we're only going to worry about four:\n\n* Number literals: `123`\n* Parentheses for grouping: `(123)`\n* Unary negation: `-123`\n* The Four Horsemen of the Arithmetic: `+`, `-`, `*`, `/`\n\nAs we work through the functions to compile each of those kinds of expressions,\nwe'll also assemble the requirements for the table-driven parser that calls\nthem.\n\n### Parsers for tokens\n\nFor now, let's focus on the Lox expressions that are each only a single token.\nIn this chapter, that's just number literals, but there will be more later. Here's\nhow we can compile them:\n\nWe map each token type to a different kind of expression. We define a function\nfor each expression that outputs the appropriate bytecode. Then we build an\narray of function pointers. The indexes in the array correspond to the\n`TokenType` enum values, and the function at each index is the code to compile\nan expression of that token type.\n\nTo compile number literals, we store a pointer to the following function at the\n`TOKEN_NUMBER` index in the array.\n\n^code number\n\nWe assume the token for the number literal has already been consumed and is\nstored in `previous`. We take that lexeme and use the C standard library to\nconvert it to a double value. Then we generate the code to load that value using\nthis function:\n\n^code emit-constant\n\nFirst, we add the value to the constant table, then we emit an `OP_CONSTANT`\ninstruction that pushes it onto the stack at runtime. To insert an entry in the\nconstant table, we rely on:\n\n^code make-constant\n\nMost of the work happens in `addConstant()`, which we defined back in an\n[earlier chapter][bytecode]. That adds the given value to the end of the chunk's\nconstant table and returns its index. The new function's job is mostly to make\nsure we don't have too many constants. Since the `OP_CONSTANT` instruction uses\na single byte for the index operand, we can store and load only up to 256 constants in a chunk.\n\n[bytecode]: chunks-of-bytecode.html\n\n\n\nThat's basically all it takes. Provided there is some suitable code that\nconsumes a `TOKEN_NUMBER` token, looks up `number()` in the function pointer\narray, and then calls it, we can now compile number literals to bytecode.\n\n### Parentheses for grouping\n\nOur as-yet-imaginary array of parsing function pointers would be great if every\nexpression was only a single token long. Alas, most are longer. However, many\nexpressions *start* with a particular token. We call these *prefix* expressions.\nFor example, when we're parsing an expression and the current token is `(`, we\nknow we must be looking at a parenthesized grouping expression.\n\nIt turns out our function pointer array handles those too. The parsing function\nfor an expression type can consume any additional tokens that it wants to, just\nlike in a regular recursive descent parser. Here's how parentheses work:\n\n^code grouping\n\nAgain, we assume the initial `(` has already been consumed. We recursively call back into `expression()` to compile the\nexpression between the parentheses, then parse the closing `)` at the end.\n\n\n\nAs far as the back end is concerned, there's literally nothing to a grouping\nexpression. Its sole function is syntactic -- it lets you insert a\nlower-precedence expression where a higher precedence is expected. Thus, it has\nno runtime semantics on its own and therefore doesn't emit any bytecode. The\ninner call to `expression()` takes care of generating bytecode for the\nexpression inside the parentheses.\n\n### Unary negation\n\nUnary minus is also a prefix expression, so it works with our model too.\n\n^code unary\n\nThe leading `-` token has been consumed and is sitting in `parser.previous`. We\ngrab the token type from that to note which unary operator we're dealing with.\nIt's unnecessary right now, but this will make more sense when we use this same\nfunction to compile the `!` operator in [the next chapter][next].\n\n[next]: types-of-values.html\n\nAs in `grouping()`, we recursively call `expression()` to compile the operand.\nAfter that, we emit the bytecode to perform the negation. It might seem a little\nweird to write the negate instruction *after* its operand's bytecode since the\n`-` appears on the left, but think about it in terms of order of execution:\n\n1. We evaluate the operand first which leaves its value on the stack.\n\n2. Then we pop that value, negate it, and push the result.\n\nSo the `OP_NEGATE` instruction should be emitted last.\nThis is part of the compiler's job -- parsing the program in the order it\nappears in the source code and rearranging it into the order that execution\nhappens.\n\n\n\nThere is one problem with this code, though. The `expression()` function it\ncalls will parse any expression for the operand, regardless of precedence. Once\nwe add binary operators and other syntax, that will do the wrong thing.\nConsider:\n\n```lox\n-a.b + c;\n```\n\nHere, the operand to `-` should be just the `a.b` expression, not the entire\n`a.b + c`. But if `unary()` calls `expression()`, the latter will happily chew\nthrough all of the remaining code including the `+`. It will erroneously treat\nthe `-` as lower precedence than the `+`.\n\nWhen parsing the operand to unary `-`, we need to compile only expressions at a\ncertain precedence level or higher. In jlox's recursive descent parser we\naccomplished that by calling into the parsing method for the lowest-precedence\nexpression we wanted to allow (in this case, `call()`). Each method for parsing\na specific expression also parsed any expressions of higher precedence too, so\nthat included the rest of the precedence table.\n\nThe parsing functions like `number()` and `unary()` here in clox are different.\nEach only parses exactly one type of expression. They don't cascade to include\nhigher-precedence expression types too. We need a different solution, and it\nlooks like this:\n\n^code parse-precedence\n\nThis function -- once we implement it -- starts at the current token and parses\nany expression at the given precedence level or higher. We have some other setup\nto get through before we can write the body of this function, but you can\nprobably guess that it will use that table of parsing function pointers I've\nbeen talking about. For now, don't worry too much about how it works. In order\nto take the \"precedence\" as a parameter, we define it numerically.\n\n^code precedence (1 before, 2 after)\n\nThese are all of Lox's precedence levels in order from lowest to highest. Since\nC implicitly gives successively larger numbers for enums, this means that\n`PREC_CALL` is numerically larger than `PREC_UNARY`. For example, say the\ncompiler is sitting on a chunk of code like:\n\n```lox\n-a.b + c\n```\n\nIf we call `parsePrecedence(PREC_ASSIGNMENT)`, then it will parse the entire\nexpression because `+` has higher precedence than assignment. If instead we\ncall `parsePrecedence(PREC_UNARY)`, it will compile the `-a.b` and stop there.\nIt doesn't keep going through the `+` because the addition has lower precedence\nthan unary operators.\n\nWith this function in hand, it's a snap to fill in the missing body for\n`expression()`.\n\n^code expression-body (1 before, 1 after)\n\nWe simply parse the lowest precedence level, which subsumes all of the\nhigher-precedence expressions too. Now, to compile the operand for a unary\nexpression, we call this new function and limit it to the appropriate level:\n\n^code unary-operand (1 before, 2 after)\n\nWe use the unary operator's own `PREC_UNARY` precedence to permit nested unary expressions like `!!doubleNegative`. Since\nunary operators have pretty high precedence, that correctly excludes things like\nbinary operators. Speaking of which...\n\n\n\n## Parsing Infix Expressions\n\nBinary operators are different from the previous expressions because they are\n*infix*. With the other expressions, we know what we are parsing from the very\nfirst token. With infix expressions, we don't know we're in the middle of a\nbinary operator until *after* we've parsed its left operand and then stumbled\nonto the operator token in the middle.\n\nHere's an example:\n\n```lox\n1 + 2\n```\n\nLet's walk through trying to compile it with what we know so far:\n\n1. We call `expression()`. That in turn calls\n `parsePrecedence(PREC_ASSIGNMENT)`.\n\n2. That function (once we implement it) sees the leading number token and\n recognizes it is parsing a number literal. It hands off control to\n `number()`.\n\n3. `number()` creates a constant, emits an `OP_CONSTANT`, and returns back to\n `parsePrecedence()`.\n\nNow what? The call to `parsePrecedence()` should consume the entire addition\nexpression, so it needs to keep going somehow. Fortunately, the parser is right\nwhere we need it to be. Now that we've compiled the leading number expression,\nthe next token is `+`. That's the exact token that `parsePrecedence()` needs to\ndetect that we're in the middle of an infix expression and to realize that the\nexpression we already compiled is actually an operand to that.\n\nSo this hypothetical array of function pointers doesn't just list functions to\nparse expressions that start with a given token. Instead, it's a *table* of\nfunction pointers. One column associates prefix parser functions with token\ntypes. The second column associates infix parser functions with token types.\n\nThe function we will use as the infix parser for `TOKEN_PLUS`, `TOKEN_MINUS`,\n`TOKEN_STAR`, and `TOKEN_SLASH` is this:\n\n^code binary\n\nWhen a prefix parser function is called, the leading token has already been\nconsumed. An infix parser function is even more *in medias res* -- the entire\nleft-hand operand expression has already been compiled and the subsequent infix\noperator consumed.\n\nThe fact that the left operand gets compiled first works out fine. It means at\nruntime, that code gets executed first. When it runs, the value it produces will\nend up on the stack. That's right where the infix operator is going to need it.\n\nThen we come here to `binary()` to handle the rest of the arithmetic operators.\nThis function compiles the right operand, much like how `unary()` compiles its\nown trailing operand. Finally, it emits the bytecode instruction that performs\nthe binary operation.\n\nWhen run, the VM will execute the left and right operand code, in that order,\nleaving their values on the stack. Then it executes the instruction for the\noperator. That pops the two values, computes the operation, and pushes the\nresult.\n\nThe code that probably caught your eye here is that `getRule()` line. When we\nparse the right-hand operand, we again need to worry about precedence. Take an\nexpression like:\n\n```lox\n2 * 3 + 4\n```\n\nWhen we parse the right operand of the `*` expression, we need to just capture\n`3`, and not `3 + 4`, because `+` is lower precedence than `*`. We could define\na separate function for each binary operator. Each would call\n`parsePrecedence()` and pass in the correct precedence level for its operand.\n\nBut that's kind of tedious. Each binary operator's right-hand operand precedence\nis one level higher than its own. We can look that up\ndynamically with this `getRule()` thing we'll get to soon. Using that, we call\n`parsePrecedence()` with one level higher than this operator's level.\n\n\n\nThis way, we can use a single `binary()` function for all binary operators even\nthough they have different precedences.\n\n## A Pratt Parser\n\nWe now have all of the pieces and parts of the compiler laid out. We have a\nfunction for each grammar production: `number()`, `grouping()`, `unary()`, and\n`binary()`. We still need to implement `parsePrecedence()`, and `getRule()`. We\nalso know we need a table that, given a token type, lets us find\n\n* the function to compile a prefix expression starting with a token of that\n type,\n\n* the function to compile an infix expression whose left operand is followed\n by a token of that type, and\n\n* the precedence of an infix expression that uses\n that token as an operator.\n\n\n\nWe wrap these three properties in a little struct which represents a single row\nin the parser table.\n\n^code parse-rule (1 before, 2 after)\n\nThat ParseFn type is a simple typedef for a function\ntype that takes no arguments and returns nothing.\n\n\n\n^code parse-fn-type (1 before, 2 after)\n\nThe table that drives our whole parser is an array of ParseRules. We've been\ntalking about it forever, and finally you get to see it.\n\n^code rules\n\n\n\nYou can see how `grouping` and `unary` are slotted into the prefix parser column\nfor their respective token types. In the next column, `binary` is wired up to\nthe four arithmetic infix operators. Those infix operators also have their\nprecedences set in the last column.\n\nAside from those, the rest of the table is full of `NULL` and `PREC_NONE`. Most\nof those empty cells are because there is no expression associated with those\ntokens. You can't start an expression with, say, `else`, and `}` would make for\na pretty confusing infix operator.\n\nBut, also, we haven't filled in the entire grammar yet. In later chapters, as we\nadd new expression types, some of these slots will get functions in them. One of\nthe things I like about this approach to parsing is that it makes it very easy\nto see which tokens are in use by the grammar and which are available.\n\nNow that we have the table, we are finally ready to write the code that uses it.\nThis is where our Pratt parser comes to life. The easiest function to define is\n`getRule()`.\n\n^code get-rule\n\nIt simply returns the rule at the given index. It's called by `binary()` to look\nup the precedence of the current operator. This function exists solely to handle\na declaration cycle in the C code. `binary()` is defined *before* the rules\ntable so that the table can store a pointer to it. That means the body of\n`binary()` cannot access the table directly.\n\nInstead, we wrap the lookup in a function. That lets us forward declare\n`getRule()` before the definition of `binary()`, and then *define* `getRule()` after the table. We'll need a\ncouple of other forward declarations to handle the fact that our grammar is\nrecursive, so let's get them all out of the way.\n\n\n\n^code forward-declarations (2 before, 1 after)\n\nIf you're following along and implementing clox yourself, pay close attention to\nthe little annotations that tell you where to put these code snippets. Don't\nworry, though, if you get it wrong, the C compiler will be happy to tell you.\n\n### Parsing with precedence\n\nNow we're getting to the fun stuff. The maestro that orchestrates all of the\nparsing functions we've defined is `parsePrecedence()`. Let's start with parsing\nprefix expressions.\n\n^code precedence-body (1 before, 1 after)\n\nWe read the next token and look up the corresponding ParseRule. If there is no\nprefix parser, then the token must be a syntax error. We report that and return\nto the caller.\n\nOtherwise, we call that prefix parse function and let it do its thing. That\nprefix parser compiles the rest of the prefix expression, consuming any other\ntokens it needs, and returns back here. Infix expressions are where it gets\ninteresting since precedence comes into play. The implementation is remarkably\nsimple.\n\n^code infix (1 before, 1 after)\n\nThat's the whole thing. Really. Here's how the entire function works: At the\nbeginning of `parsePrecedence()`, we look up a prefix parser for the current\ntoken. The first token is *always* going to belong to some kind of prefix\nexpression, by definition. It may turn out to be nested as an operand inside one\nor more infix expressions, but as you read the code from left to right, the\nfirst token you hit always belongs to a prefix expression.\n\nAfter parsing that, which may consume more tokens, the prefix expression is\ndone. Now we look for an infix parser for the next token. If we find one, it\nmeans the prefix expression we already compiled might be an operand for it. But\nonly if the call to `parsePrecedence()` has a `precedence` that is low enough to\npermit that infix operator.\n\nIf the next token is too low precedence, or isn't an infix operator at all,\nwe're done. We've parsed as much expression as we can. Otherwise, we consume the\noperator and hand off control to the infix parser we found. It consumes whatever\nother tokens it needs (usually the right operand) and returns back to\n`parsePrecedence()`. Then we loop back around and see if the *next* token is\nalso a valid infix operator that can take the entire preceding expression as its\noperand. We keep looping like that, crunching through infix operators and their\noperands until we hit a token that isn't an infix operator or is too low\nprecedence and stop.\n\nThat's a lot of prose, but if you really want to mind meld with Vaughan Pratt\nand fully understand the algorithm, step through the parser in your debugger as\nit works through some expressions. Maybe a picture will help. There's only a\nhandful of functions, but they are marvelously intertwined:\n\n\n\n\"The\n\n\n\nLater, we'll need to tweak the code in this chapter to handle assignment. But,\notherwise, what we wrote covers all of our expression compiling needs for the\nrest of the book. We'll plug additional parsing functions into the table when we\nadd new kinds of expressions, but `parsePrecedence()` is complete.\n\n## Dumping Chunks\n\nWhile we're here in the core of our compiler, we should put in some\ninstrumentation. To help debug the generated bytecode, we'll add support for\ndumping the chunk once the compiler finishes. We had some temporary logging\nearlier when we hand-authored the chunk. Now we'll put in some real code so that\nwe can enable it whenever we want.\n\nSince this isn't for end users, we hide it behind a flag.\n\n^code define-debug-print-code (2 before, 1 after)\n\nWhen that flag is defined, we use our existing \"debug\" module to print out the\nchunk's bytecode.\n\n^code dump-chunk (1 before, 1 after)\n\nWe do this only if the code was free of errors. After a syntax error, the\ncompiler keeps on going but it's in kind of a weird state and might produce\nbroken code. That's harmless because it won't get executed, but we'll just\nconfuse ourselves if we try to read it.\n\nFinally, to access `disassembleChunk()`, we need to include its header.\n\n^code include-debug (1 before, 2 after)\n\nWe made it! This was the last major section to install in our VM's compilation\nand execution pipeline. Our interpreter doesn't *look* like much, but inside it\nis scanning, parsing, compiling to bytecode, and executing.\n\nFire up the VM and type in an expression. If we did everything right, it should\ncalculate and print the result. We now have a very over-engineered arithmetic\ncalculator. We have a lot of language features to add in the coming chapters,\nbut the foundation is in place.\n\n
\n\n## Challenges\n\n1. To really understand the parser, you need to see how execution threads\n through the interesting parsing functions -- `parsePrecedence()` and the\n parser functions stored in the table. Take this (strange) expression:\n\n ```lox\n (-1 + 2) * 3 - -4\n ```\n\n Write a trace of how those functions are called. Show the order they are\n called, which calls which, and the arguments passed to them.\n\n2. The ParseRule row for `TOKEN_MINUS` has both prefix and infix function\n pointers. That's because `-` is both a prefix operator (unary negation) and\n an infix one (subtraction).\n\n In the full Lox language, what other tokens can be used in both prefix and\n infix positions? What about in C or in another language of your choice?\n\n3. You might be wondering about complex \"mixfix\" expressions that have more\n than two operands separated by tokens. C's conditional or \"ternary\"\n operator, `?:`, is a widely known one.\n\n Add support for that operator to the compiler. You don't have to generate\n any bytecode, just show how you would hook it up to the parser and handle\n the operands.\n\n
\n\n
\n\n## Design Note: It's Just Parsing\n\nI'm going to make a claim here that will be unpopular with some compiler and\nlanguage people. It's OK if you don't agree. Personally, I learn more from\nstrongly stated opinions that I disagree with than I do from several pages of\nqualifiers and equivocation. My claim is that *parsing doesn't matter*.\n\nOver the years, many programming language people, especially in academia, have\ngotten *really* into parsers and taken them very seriously. Initially, it was\nthe compiler folks who got into compiler-compilers,\nLALR, and other stuff like that. The first half of the dragon book is a long\nlove letter to the wonders of parser generators.\n\n\n\nLater, the functional programming folks got into parser combinators, packrat\nparsers, and other sorts of things. Because, obviously, if you give a functional\nprogrammer a problem, the first thing they'll do is whip out a pocketful of\nhigher-order functions.\n\nOver in math and algorithm analysis land, there is a long legacy of research\ninto proving time and memory usage for various parsing techniques, transforming\nparsing problems into other problems and back, and assigning complexity classes\nto different grammars.\n\nAt one level, this stuff is important. If you're implementing a language, you\nwant some assurance that your parser won't go exponential and take 7,000 years\nto parse a weird edge case in the grammar. Parser theory gives you that bound.\nAs an intellectual exercise, learning about parsing techniques is also fun and\nrewarding.\n\nBut if your goal is just to implement a language and get it in front of users,\nalmost all of that stuff doesn't matter. It's really easy to get worked up by\nthe enthusiasm of the people who *are* into it and think that your front end\n*needs* some whiz-bang generated combinator-parser-factory thing. I've seen\npeople burn tons of time writing and rewriting their parser using whatever\ntoday's hot library or technique is.\n\nThat's time that doesn't add any value to your user's life. If you're just\ntrying to get your parser done, pick one of the bog-standard techniques, use it,\nand move on. Recursive descent, Pratt parsing, and the popular parser generators\nlike ANTLR or Bison are all fine.\n\nTake the extra time you saved not rewriting your parsing code and spend it\nimproving the compile error messages your compiler shows users. Good error\nhandling and reporting is more valuable to users than almost anything else you\ncan put time into in the front end.\n\n
\n" }, { "path": "book/contents.md", "content": "This text is not used. All of the content is in the contents.html template.\n" }, { "path": "book/control-flow.md", "content": "> Logic, like whiskey, loses its beneficial effect when taken in too large\n> quantities.\n>\n> Edward John Moreton Drax Plunkett, Lord Dunsany\n\nCompared to [last chapter's][statements] grueling marathon, today is a\nlighthearted frolic through a daisy meadow. But while the work is easy, the\nreward is surprisingly large.\n\n[statements]: statements-and-state.html\n\nRight now, our interpreter is little more than a calculator. A Lox program can\nonly do a fixed amount of work before completing. To make it run twice as long\nyou have to make the source code twice as lengthy. We're about to fix that. In\nthis chapter, our interpreter takes a big step towards the programming\nlanguage major leagues: *Turing-completeness*.\n\n## Turing Machines (Briefly)\n\nIn the early part of last century, mathematicians stumbled into a series of\nconfusing paradoxes that led them to doubt the\nstability of the foundation they had built their work upon. To address that\n[crisis][], they went back to square one. Starting from a handful of axioms,\nlogic, and set theory, they hoped to rebuild mathematics on top of an\nimpervious foundation.\n\n[crisis]: https://en.wikipedia.org/wiki/Foundations_of_mathematics#Foundational_crisis\n\n\n\nThey wanted to rigorously answer questions like, \"Can all true statements be\nproven?\", \"Can we [compute][] all functions that we can define?\", or even the\nmore general question, \"What do we mean when we claim a function is\n'computable'?\"\n\n[compute]: https://en.wikipedia.org/wiki/Computable_function\n\nThey presumed the answer to the first two questions would be \"yes\". All that\nremained was to prove it. It turns out that the answer to both is \"no\", and\nastonishingly, the two questions are deeply intertwined. This is a fascinating\ncorner of mathematics that touches fundamental questions about what brains are\nable to do and how the universe works. I can't do it justice here.\n\nWhat I do want to note is that in the process of proving that the answer to the\nfirst two questions is \"no\", Alan Turing and Alonzo Church devised a precise\nanswer to the last question -- a definition of what kinds of functions are computable. They each crafted a tiny system with a\nminimum set of machinery that is still powerful enough to compute any of a\n(very) large class of functions.\n\n\n\nThese are now considered the \"computable functions\". Turing's system is called a\n**Turing machine**. Church's is the **lambda\ncalculus**. Both are still widely used as the basis for models of computation\nand, in fact, many modern functional programming languages use the lambda\ncalculus at their core.\n\n\n\n\"A\n\nTuring machines have better name recognition -- there's no Hollywood film about\nAlonzo Church yet -- but the two formalisms are [equivalent in power][thesis].\nIn fact, any programming language with some minimal level of expressiveness is\npowerful enough to compute *any* computable function.\n\n[thesis]: https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis\n\nYou can prove that by writing a simulator for a Turing machine in your language.\nSince Turing proved his machine can compute any computable function, by\nextension, that means your language can too. All you need to do is translate the\nfunction into a Turing machine, and then run that on your simulator.\n\nIf your language is expressive enough to do that, it's considered\n**Turing-complete**. Turing machines are pretty dang simple, so it doesn't take\nmuch power to do this. You basically need arithmetic, a little control flow,\nand the ability to allocate and use (theoretically) arbitrary amounts of memory.\nWe've got the first. By the end of this chapter, we'll have the second.\n\n\n\n## Conditional Execution\n\nEnough history, let's jazz up our language. We can divide control flow roughly\ninto two kinds:\n\n* **Conditional** or **branching control flow** is used to *not* execute\n some piece of code. Imperatively, you can think of it as jumping *ahead*\n over a region of code.\n\n* **Looping control flow** executes a chunk of code more than once. It jumps\n *back* so that you can do something again. Since you don't usually want\n *infinite* loops, it typically has some conditional logic to know when to\n stop looping as well.\n\nBranching is simpler, so we'll start there. C-derived languages have two main\nconditional execution features, the `if` statement and the perspicaciously named\n\"conditional\" operator (`?:`). An `if` statement\nlets you conditionally execute statements and the conditional operator lets you\nconditionally execute expressions.\n\n\n\nFor simplicity's sake, Lox doesn't have a conditional operator, so let's get our\n`if` statement on. Our statement grammar gets a new production.\n\n\n\n```ebnf\nstatement → exprStmt\n | ifStmt\n | printStmt\n | block ;\n\nifStmt → \"if\" \"(\" expression \")\" statement\n ( \"else\" statement )? ;\n```\n\n\n\nAn `if` statement has an expression for the condition, then a statement to execute\nif the condition is truthy. Optionally, it may also have an `else` keyword and a\nstatement to execute if the condition is falsey. The syntax\ntree node has fields for each of those three pieces.\n\n^code if-ast (1 before, 1 after)\n\n\n\nLike other statements, the parser recognizes an `if` statement by the leading\n`if` keyword.\n\n^code match-if (1 before, 1 after)\n\nWhen it finds one, it calls this new method to parse the rest:\n\n^code if-statement\n\n\n\nAs usual, the parsing code hews closely to the grammar. It detects an else\nclause by looking for the preceding `else` keyword. If there isn't one, the\n`elseBranch` field in the syntax tree is `null`.\n\nThat seemingly innocuous optional else has, in fact, opened up an ambiguity in\nour grammar. Consider:\n\n```lox\nif (first) if (second) whenTrue(); else whenFalse();\n```\n\nHere's the riddle: Which `if` statement does that else clause belong to? This\nisn't just a theoretical question about how we notate our grammar. It actually\naffects how the code executes:\n\n* If we attach the else to the first `if` statement, then `whenFalse()` is\n called if `first` is falsey, regardless of what value `second` has.\n\n* If we attach it to the second `if` statement, then `whenFalse()` is only\n called if `first` is truthy and `second` is falsey.\n\nSince else clauses are optional, and there is no explicit delimiter marking the\nend of the `if` statement, the grammar is ambiguous when you nest `if`s in this\nway. This classic pitfall of syntax is called the **[dangling else][]** problem.\n\n[dangling else]: https://en.wikipedia.org/wiki/Dangling_else\n\n\n\n\"Two\n\n\n\nIt *is* possible to define a context-free grammar that avoids the ambiguity\ndirectly, but it requires splitting most of the statement rules into pairs, one\nthat allows an `if` with an `else` and one that doesn't. It's annoying.\n\nInstead, most languages and parsers avoid the problem in an ad hoc way. No\nmatter what hack they use to get themselves out of the trouble, they always\nchoose the same interpretation -- the `else` is bound to the nearest `if` that\nprecedes it.\n\nOur parser conveniently does that already. Since `ifStatement()` eagerly looks\nfor an `else` before returning, the innermost call to a nested series will claim\nthe else clause for itself before returning to the outer `if` statements.\n\nSyntax in hand, we are ready to interpret.\n\n^code visit-if\n\nThe interpreter implementation is a thin wrapper around the self-same Java code.\nIt evaluates the condition. If truthy, it executes the then branch. Otherwise,\nif there is an else branch, it executes that.\n\nIf you compare this code to how the interpreter handles other syntax we've\nimplemented, the part that makes control flow special is that Java `if`\nstatement. Most other syntax trees always evaluate their subtrees. Here, we may\nnot evaluate the then or else statement. If either of those has a side effect,\nthe choice not to evaluate it becomes user visible.\n\n## Logical Operators\n\nSince we don't have the conditional operator, you might think we're done with\nbranching, but no. Even without the ternary operator, there are two other\noperators that are technically control flow constructs -- the logical operators\n`and` and `or`.\n\nThese aren't like other binary operators because they **short-circuit**. If,\nafter evaluating the left operand, we know what the result of the logical\nexpression must be, we don't evaluate the right operand. For example:\n\n```lox\nfalse and sideEffect();\n```\n\nFor an `and` expression to evaluate to something truthy, both operands must be\ntruthy. We can see as soon as we evaluate the left `false` operand that that\nisn't going to be the case, so there's no need to evaluate `sideEffect()` and it\ngets skipped.\n\nThis is why we didn't implement the logical operators with the other binary\noperators. Now we're ready. The two new operators are low in the precedence\ntable. Similar to `||` and `&&` in C, they each have their own precedence with `or` lower than `and`. We slot them\nright between `assignment` and `equality`.\n\n\n\n```ebnf\nexpression → assignment ;\nassignment → IDENTIFIER \"=\" assignment\n | logic_or ;\nlogic_or → logic_and ( \"or\" logic_and )* ;\nlogic_and → equality ( \"and\" equality )* ;\n```\n\nInstead of falling back to `equality`, `assignment` now cascades to `logic_or`.\nThe two new rules, `logic_or` and `logic_and`, are similar to other binary operators. Then `logic_and` calls\nout to `equality` for its operands, and we chain back to the rest of the\nexpression rules.\n\n\n\nWe could reuse the existing Expr.Binary class for these two new expressions\nsince they have the same fields. But then `visitBinaryExpr()` would have to\ncheck to see if the operator is one of the logical operators and use a different\ncode path to handle the short circuiting. I think it's cleaner to define a new class for these operators so that they get their\nown visit method.\n\n^code logical-ast (1 before, 1 after)\n\n\n\nTo weave the new expressions into the parser, we first change the parsing code\nfor assignment to call `or()`.\n\n^code or-in-assignment (1 before, 2 after)\n\nThe code to parse a series of `or` expressions mirrors other binary operators.\n\n^code or\n\nIts operands are the next higher level of precedence, the new `and` expression.\n\n^code and\n\nThat calls `equality()` for its operands, and with that, the expression parser\nis all tied back together again. We're ready to interpret.\n\n^code visit-logical\n\nIf you compare this to the [earlier chapter's][evaluating] `visitBinaryExpr()`\nmethod, you can see the difference. Here, we evaluate the left operand first. We\nlook at its value to see if we can short-circuit. If not, and only then, do we\nevaluate the right operand.\n\n[evaluating]: evaluating-expressions.html\n\nThe other interesting piece here is deciding what actual value to return. Since\nLox is dynamically typed, we allow operands of any type and use truthiness to\ndetermine what each operand represents. We apply similar reasoning to the\nresult. Instead of promising to literally return `true` or `false`, a logic\noperator merely guarantees it will return a value with appropriate truthiness.\n\nFortunately, we have values with proper truthiness right at hand -- the results\nof the operands themselves. So we use those. For example:\n\n```lox\nprint \"hi\" or 2; // \"hi\".\nprint nil or \"yes\"; // \"yes\".\n```\n\nOn the first line, `\"hi\"` is truthy, so the `or` short-circuits and returns\nthat. On the second line, `nil` is falsey, so it evaluates and returns the\nsecond operand, `\"yes\"`.\n\nThat covers all of the branching primitives in Lox. We're ready to jump ahead to\nloops. You see what I did there? *Jump. Ahead.* Get it? See, it's like a\nreference to... oh, forget it.\n\n## While Loops\n\nLox features two looping control flow statements, `while` and `for`. The `while`\nloop is the simpler one, so we'll start there. Its grammar is the same as in C.\n\n```ebnf\nstatement → exprStmt\n | ifStmt\n | printStmt\n | whileStmt\n | block ;\n\nwhileStmt → \"while\" \"(\" expression \")\" statement ;\n```\n\nWe add another clause to the statement rule that points to the new rule for\nwhile. It takes a `while` keyword, followed by a parenthesized condition\nexpression, then a statement for the body. That new grammar rule gets a syntax tree node.\n\n^code while-ast (1 before, 1 after)\n\n\n\nThe node stores the condition and body. Here you can see why it's nice to have\nseparate base classes for expressions and statements. The field declarations\nmake it clear that the condition is an expression and the body is a statement.\n\nOver in the parser, we follow the same process we used for `if` statements.\nFirst, we add another case in `statement()` to detect and match the leading\nkeyword.\n\n^code match-while (1 before, 1 after)\n\nThat delegates the real work to this method:\n\n^code while-statement\n\nThe grammar is dead simple and this is a straight translation of it to Java.\nSpeaking of translating straight to Java, here's how we execute the new syntax:\n\n^code visit-while\n\nLike the visit method for `if`, this visitor uses the corresponding Java\nfeature. This method isn't complex, but it makes Lox much more powerful. We can\nfinally write a program whose running time isn't strictly bound by the length of\nthe source code.\n\n## For Loops\n\nWe're down to the last control flow construct, Ye Olde\nC-style `for` loop. I probably don't need to remind you, but it looks like this:\n\n```lox\nfor (var i = 0; i < 10; i = i + 1) print i;\n```\n\nIn grammarese, that's:\n\n```ebnf\nstatement → exprStmt\n | forStmt\n | ifStmt\n | printStmt\n | whileStmt\n | block ;\n\nforStmt → \"for\" \"(\" ( varDecl | exprStmt | \";\" )\n expression? \";\"\n expression? \")\" statement ;\n```\n\n\n\nInside the parentheses, you have three clauses separated by semicolons:\n\n1. The first clause is the *initializer*. It is executed exactly once, before\n anything else. It's usually an expression, but for convenience, we also\n allow a variable declaration. In that case, the variable is scoped to the\n rest of the `for` loop -- the other two clauses and the body.\n\n2. Next is the *condition*. As in a `while` loop, this expression controls when\n to exit the loop. It's evaluated once at the beginning of each iteration,\n including the first. If the result is truthy, it executes the loop body.\n Otherwise, it bails.\n\n3. The last clause is the *increment*. It's an arbitrary expression that does\n some work at the end of each loop iteration. The result of the expression is\n discarded, so it must have a side effect to be useful. In practice, it\n usually increments a variable.\n\nAny of these clauses can be omitted. Following the closing parenthesis is a\nstatement for the body, which is typically a block.\n\n### Desugaring\n\nThat's a lot of machinery, but note that none of it does anything you couldn't\ndo with the statements we already have. If `for` loops didn't support\ninitializer clauses, you could just put the initializer expression before the\n`for` statement. Without an increment clause, you could simply put the increment\nexpression at the end of the body yourself.\n\nIn other words, Lox doesn't *need* `for` loops, they just make some common code\npatterns more pleasant to write. These kinds of features are called **syntactic sugar**. For example, the previous `for` loop\ncould be rewritten like so:\n\n\n\n```lox\n{\n var i = 0;\n while (i < 10) {\n print i;\n i = i + 1;\n }\n}\n```\n\nThis script has the exact same semantics as the previous one, though it's not as\neasy on the eyes. Syntactic sugar features like Lox's `for` loop make a language\nmore pleasant and productive to work in. But, especially in sophisticated\nlanguage implementations, every language feature that requires back-end support\nand optimization is expensive.\n\nWe can have our cake and eat it too by **desugaring**. That funny word describes a process where\nthe front end takes code using syntax sugar and translates it to a more\nprimitive form that the back end already knows how to execute.\n\n\n\nWe're going to desugar `for` loops to the `while` loops and other statements the\ninterpreter already handles. In our simple interpreter, desugaring really\ndoesn't save us much work, but it does give me an excuse to introduce you to the\ntechnique. So, unlike the previous statements, we *won't* add a new syntax tree\nnode. Instead, we go straight to parsing. First, add an import we'll need soon.\n\n^code import-arrays (1 before, 1 after)\n\nLike every statement, we start parsing a `for` loop by matching its keyword.\n\n^code match-for (1 before, 1 after)\n\nHere is where it gets interesting. The desugaring is going to happen here, so\nwe'll build this method a piece at a time, starting with the opening parenthesis\nbefore the clauses.\n\n^code for-statement\n\nThe first clause following that is the initializer.\n\n^code for-initializer (2 before, 1 after)\n\nIf the token following the `(` is a semicolon then the initializer has been\nomitted. Otherwise, we check for a `var` keyword to see if it's a variable declaration. If neither of those matched, it\nmust be an expression. We parse that and wrap it in an expression statement so\nthat the initializer is always of type Stmt.\n\n\n\nNext up is the condition.\n\n^code for-condition (2 before, 1 after)\n\nAgain, we look for a semicolon to see if the clause has been omitted. The last\nclause is the increment.\n\n^code for-increment (1 before, 1 after)\n\nIt's similar to the condition clause except this one is terminated by the\nclosing parenthesis. All that remains is the body.\n\n\n\n^code for-body (1 before, 1 after)\n\nWe've parsed all of the various pieces of the `for` loop and the resulting AST\nnodes are sitting in a handful of Java local variables. This is where the\ndesugaring comes in. We take those and use them to synthesize syntax tree nodes\nthat express the semantics of the `for` loop, like the hand-desugared example I\nshowed you earlier.\n\nThe code is a little simpler if we work backward, so we start with the increment\nclause.\n\n^code for-desugar-increment (2 before, 1 after)\n\nThe increment, if there is one, executes after the body in each iteration of the\nloop. We do that by replacing the body with a little block that contains the\noriginal body followed by an expression statement that evaluates the increment.\n\n^code for-desugar-condition (2 before, 1 after)\n\nNext, we take the condition and the body and build the loop using a primitive\n`while` loop. If the condition is omitted, we jam in `true` to make an infinite\nloop.\n\n^code for-desugar-initializer (2 before, 1 after)\n\nFinally, if there is an initializer, it runs once before the entire loop. We do\nthat by, again, replacing the whole statement with a block that runs the\ninitializer and then executes the loop.\n\nThat's it. Our interpreter now supports C-style `for` loops and we didn't have\nto touch the Interpreter class at all. Since we desugared to nodes the\ninterpreter already knows how to visit, there is no more work to do.\n\nFinally, Lox is powerful enough to entertain us, at least for a few minutes.\nHere's a tiny program to print the first 21 elements in the Fibonacci\nsequence:\n\n```lox\nvar a = 0;\nvar temp;\n\nfor (var b = 1; a < 10000; b = temp + b) {\n print a;\n temp = a;\n a = b;\n}\n```\n\n
\n\n## Challenges\n\n1. A few chapters from now, when Lox supports first-class functions and dynamic\n dispatch, we technically won't *need* branching statements built into the\n language. Show how conditional execution can be implemented in terms of\n those. Name a language that uses this technique for its control flow.\n\n2. Likewise, looping can be implemented using those same tools, provided our\n interpreter supports an important optimization. What is it, and why is it\n necessary? Name a language that uses this technique for iteration.\n\n3. Unlike Lox, most other C-style languages also support `break` and `continue`\n statements inside loops. Add support for `break` statements.\n\n The syntax is a `break` keyword followed by a semicolon. It should be a\n syntax error to have a `break` statement appear outside of any enclosing\n loop. At runtime, a `break` statement causes execution to jump to the end of\n the nearest enclosing loop and proceeds from there. Note that the `break`\n may be nested inside other blocks and `if` statements that also need to be\n exited.\n\n
\n\n
\n\n## Design Note: Spoonfuls of Syntactic Sugar\n\nWhen you design your own language, you choose how much syntactic sugar to pour\ninto the grammar. Do you make an unsweetened health food where each semantic\noperation maps to a single syntactic unit, or some decadent dessert where every\nbit of behavior can be expressed ten different ways? Successful languages\ninhabit all points along this continuum.\n\nOn the extreme acrid end are those with ruthlessly minimal syntax like Lisp,\nForth, and Smalltalk. Lispers famously claim their language \"has no syntax\",\nwhile Smalltalkers proudly show that you can fit the entire grammar on an index\ncard. This tribe has the philosophy that the *language* doesn't need syntactic\nsugar. Instead, the minimal syntax and semantics it provides are powerful enough\nto let library code be as expressive as if it were part of the language itself.\n\nNear these are languages like C, Lua, and Go. They aim for simplicity and\nclarity over minimalism. Some, like Go, deliberately eschew both syntactic sugar\nand the kind of syntactic extensibility of the previous category. They want the\nsyntax to get out of the way of the semantics, so they focus on keeping both the\ngrammar and libraries simple. Code should be obvious more than beautiful.\n\nSomewhere in the middle you have languages like Java, C#, and Python. Eventually\nyou reach Ruby, C++, Perl, and D -- languages which have stuffed so much syntax\ninto their grammar, they are running out of punctuation characters on the\nkeyboard.\n\nTo some degree, location on the spectrum correlates with age. It's relatively\neasy to add bits of syntactic sugar in later releases. New syntax is a crowd\npleaser, and it's less likely to break existing programs than mucking with the\nsemantics. Once added, you can never take it away, so languages tend to sweeten\nwith time. One of the main benefits of creating a new language from scratch is\nit gives you an opportunity to scrape off those accumulated layers of frosting\nand start over.\n\nSyntactic sugar has a bad rap among the PL intelligentsia. There's a real fetish\nfor minimalism in that crowd. There is some justification for that. Poorly\ndesigned, unneeded syntax raises the cognitive load without adding enough\nexpressiveness to carry its weight. Since there is always pressure to cram new\nfeatures into the language, it takes discipline and a focus on simplicity to\navoid bloat. Once you add some syntax, you're stuck with it, so it's smart to be\nparsimonious.\n\nAt the same time, most successful languages do have fairly complex grammars, at\nleast by the time they are widely used. Programmers spend a ton of time in their\nlanguage of choice, and a few niceties here and there really can improve the\ncomfort and efficiency of their work.\n\nStriking the right balance -- choosing the right level of sweetness for your\nlanguage -- relies on your own sense of taste.\n\n
\n" }, { "path": "book/dedication.md", "content": "
\n\n\"My\n\nTo Ginny, I miss your stupid face.\n\n
" }, { "path": "book/evaluating-expressions.md", "content": "> You are my creator, but I am your master; Obey!\n>\n> Mary Shelley, Frankenstein\n\nIf you want to properly set the mood for this chapter, try to conjure up a\nthunderstorm, one of those swirling tempests that likes to yank open shutters at\nthe climax of the story. Maybe toss in a few bolts of lightning. In this\nchapter, our interpreter will take breath, open its eyes, and execute some code.\n\n\n\n\"A\n\n\n\nThere are all manner of ways that language implementations make a computer do\nwhat the user's source code commands. They can compile it to machine code,\ntranslate it to another high-level language, or reduce it to some bytecode\nformat for a virtual machine to run. For our first interpreter, though, we are\ngoing to take the simplest, shortest path and execute the syntax tree itself.\n\nRight now, our parser only supports expressions. So, to \"execute\" code, we will\nevaluate an expression and produce a value. For each kind of expression syntax\nwe can parse -- literal, operator, etc. -- we need a corresponding chunk of code\nthat knows how to evaluate that tree and produce a result. That raises two\nquestions:\n\n1. What kinds of values do we produce?\n\n2. How do we organize those chunks of code?\n\nTaking them on one at a time...\n\n## Representing Values\n\nIn Lox, values are created by literals, computed by\nexpressions, and stored in variables. The user sees these as *Lox* objects, but\nthey are implemented in the underlying language our interpreter is written in.\nThat means bridging the lands of Lox's dynamic typing and Java's static types. A\nvariable in Lox can store a value of any (Lox) type, and can even store values\nof different types at different points in time. What Java type might we use to\nrepresent that?\n\n\n\nGiven a Java variable with that static type, we must also be able to determine\nwhich kind of value it holds at runtime. When the interpreter executes a `+`\noperator, it needs to tell if it is adding two numbers or concatenating two\nstrings. Is there a Java type that can hold numbers, strings, Booleans, and\nmore? Is there one that can tell us what its runtime type is? There is! Good old\njava.lang.Object.\n\nIn places in the interpreter where we need to store a Lox value, we can use\nObject as the type. Java has boxed versions of its primitive types that all\nsubclass Object, so we can use those for Lox's built-in types:\n\n\n\n\n \n \n\n\n\n\n \n \n\n\n \n \n\n\n \n \n\n\n \n \n\n\n \n \n\n\n
Lox typeJava representation
Any Lox valueObject
nilnull
BooleanBoolean
numberDouble
stringString
\n\nGiven a value of static type Object, we can determine if the runtime value is a\nnumber or a string or whatever using Java's built-in `instanceof` operator. In\nother words, the JVM's own object representation\nconveniently gives us everything we need to implement Lox's built-in types.\nWe'll have to do a little more work later when we add Lox's notions of\nfunctions, classes, and instances, but Object and the boxed primitive classes\nare sufficient for the types we need right now.\n\n\n\n## Evaluating Expressions\n\nNext, we need blobs of code to implement the evaluation logic for each kind of\nexpression we can parse. We could stuff that code into the syntax tree classes\nin something like an `interpret()` method. In effect, we could tell each syntax\ntree node, \"Interpret thyself\". This is the Gang of Four's\n[Interpreter design pattern][]. It's a neat pattern, but like I mentioned\nearlier, it gets messy if we jam all sorts of logic into the tree classes.\n\n[interpreter design pattern]: https://en.wikipedia.org/wiki/Interpreter_pattern\n\nInstead, we're going to reuse our groovy [Visitor pattern][]. In the previous\nchapter, we created an AstPrinter class. It took in a syntax tree and\nrecursively traversed it, building up a string which it ultimately returned.\nThat's almost exactly what a real interpreter does, except instead of\nconcatenating strings, it computes values.\n\n[visitor pattern]: representing-code.html#the-visitor-pattern\n\nWe start with a new class.\n\n^code interpreter-class\n\nThe class declares that it's a visitor. The return type of the visit methods\nwill be Object, the root class that we use to refer to a Lox value in our Java\ncode. To satisfy the Visitor interface, we need to define visit methods for each\nof the four expression tree classes our parser produces. We'll start with the\nsimplest...\n\n### Evaluating literals\n\nThe leaves of an expression tree -- the atomic bits of syntax that all other\nexpressions are composed of -- are literals. Literals\nare almost values already, but the distinction is important. A literal is a *bit\nof syntax* that produces a value. A literal always appears somewhere in the\nuser's source code. Lots of values are produced by computation and don't exist\nanywhere in the code itself. Those aren't literals. A literal comes from the\nparser's domain. Values are an interpreter concept, part of the runtime's world.\n\n\n\nSo, much like we converted a literal *token* into a literal *syntax tree node*\nin the parser, now we convert the literal tree node into a runtime value. That\nturns out to be trivial.\n\n^code visit-literal\n\nWe eagerly produced the runtime value way back during scanning and stuffed it in\nthe token. The parser took that value and stuck it in the literal tree node,\nso to evaluate a literal, we simply pull it back out.\n\n### Evaluating parentheses\n\nThe next simplest node to evaluate is grouping -- the node you get as a result\nof using explicit parentheses in an expression.\n\n^code visit-grouping\n\nA grouping node has a reference to an inner node\nfor the expression contained inside the parentheses. To evaluate the grouping\nexpression itself, we recursively evaluate that subexpression and return it.\n\nWe rely on this helper method which simply sends the expression back into the\ninterpreter's visitor implementation:\n\n\n\n^code evaluate\n\n### Evaluating unary expressions\n\nLike grouping, unary expressions have a single subexpression that we must\nevaluate first. The difference is that the unary expression itself does a little\nwork afterwards.\n\n^code visit-unary\n\nFirst, we evaluate the operand expression. Then we apply the unary operator\nitself to the result of that. There are two different unary expressions,\nidentified by the type of the operator token.\n\nShown here is `-`, which negates the result of the subexpression. The\nsubexpression must be a number. Since we don't *statically* know that in Java,\nwe cast it before performing the operation. This type\ncast happens at runtime when the `-` is evaluated. That's the core of what makes\na language dynamically typed right there.\n\n\n\nYou can start to see how evaluation recursively traverses the tree. We can't\nevaluate the unary operator itself until after we evaluate its operand\nsubexpression. That means our interpreter is doing a **post-order traversal** --\neach node evaluates its children before doing its own work.\n\nThe other unary operator is logical not.\n\n^code unary-bang (1 before, 1 after)\n\nThe implementation is simple, but what is this \"truthy\" thing about? We need to\nmake a little side trip to one of the great questions of Western philosophy:\n*What is truth?*\n\n### Truthiness and falsiness\n\nOK, maybe we're not going to really get into the universal question, but at\nleast inside the world of Lox, we need to decide what happens when you use\nsomething other than `true` or `false` in a logic operation like `!` or any\nother place where a Boolean is expected.\n\nWe *could* just say it's an error because we don't roll with implicit\nconversions, but most dynamically typed languages aren't that ascetic. Instead,\nthey take the universe of values of all types and partition them into two sets,\none of which they define to be \"true\", or \"truthful\", or (my favorite) \"truthy\",\nand the rest which are \"false\" or \"falsey\". This partitioning is somewhat\narbitrary and gets weird in a few languages.\n\n\n\nLox follows Ruby's simple rule: `false` and `nil` are falsey, and everything else\nis truthy. We implement that like so:\n\n^code is-truthy\n\n### Evaluating binary operators\n\nOn to the last expression tree class, binary operators. There's a handful of\nthem, and we'll start with the arithmetic ones.\n\n^code visit-binary\n\n\n\nI think you can figure out what's going on here. The main difference from the\nunary negation operator is that we have two operands to evaluate.\n\nI left out one arithmetic operator because it's a little special.\n\n^code binary-plus (3 before, 1 after)\n\nThe `+` operator can also be used to concatenate two strings. To handle that, we\ndon't just assume the operands are a certain type and *cast* them, we\ndynamically *check* the type and choose the appropriate operation. This is why\nwe need our object representation to support `instanceof`.\n\n\n\nNext up are the comparison operators.\n\n^code binary-comparison (1 before, 1 after)\n\nThey are basically the same as arithmetic. The only difference is that where the\narithmetic operators produce a value whose type is the same as the operands\n(numbers or strings), the comparison operators always produce a Boolean.\n\nThe last pair of operators are equality.\n\n^code binary-equality\n\nUnlike the comparison operators which require numbers, the equality operators\nsupport operands of any type, even mixed ones. You can't ask Lox if 3 is *less*\nthan `\"three\"`, but you can ask if it's *equal* to\nit.\n\n\n\nLike truthiness, the equality logic is hoisted out into a separate method.\n\n^code is-equal\n\nThis is one of those corners where the details of how we represent Lox objects\nin terms of Java matter. We need to correctly implement *Lox's* notion of\nequality, which may be different from Java's.\n\nFortunately, the two are pretty similar. Lox doesn't do implicit conversions in\nequality and Java does not either. We do have to handle `nil`/`null` specially\nso that we don't throw a NullPointerException if we try to call `equals()` on\n`null`. Otherwise, we're fine. Java's `equals()` method\non Boolean, Double, and String have the behavior we want for Lox.\n\n\n\nAnd that's it! That's all the code we need to correctly interpret a valid Lox\nexpression. But what about an *invalid* one? In particular, what happens when a\nsubexpression evaluates to an object of the wrong type for the operation being\nperformed?\n\n## Runtime Errors\n\nI was cavalier about jamming casts in whenever a subexpression produces an\nObject and the operator requires it to be a number or a string. Those casts can\nfail. Even though the user's code is erroneous, if we want to make a usable language, we are responsible for handling that error\ngracefully.\n\n\n\nIt's time for us to talk about **runtime errors**. I spilled a lot of ink in the\nprevious chapters talking about error handling, but those were all *syntax* or\n*static* errors. Those are detected and reported before *any* code is executed.\nRuntime errors are failures that the language semantics demand we detect and\nreport while the program is running (hence the name).\n\nRight now, if an operand is the wrong type for the operation being performed,\nthe Java cast will fail and the JVM will throw a ClassCastException. That\nunwinds the whole stack and exits the application, vomiting a Java stack trace\nonto the user. That's probably not what we want. The fact that Lox is\nimplemented in Java should be a detail hidden from the user. Instead, we want\nthem to understand that a *Lox* runtime error occurred, and give them an error\nmessage relevant to our language and their program.\n\nThe Java behavior does have one thing going for it, though. It correctly stops\nexecuting any code when the error occurs. Let's say the user enters some\nexpression like:\n\n```lox\n2 * (3 / -\"muffin\")\n```\n\nYou can't negate a muffin, so we need to report a\nruntime error at that inner `-` expression. That in turn means we can't evaluate\nthe `/` expression since it has no meaningful right operand. Likewise for the\n`*`. So when a runtime error occurs deep in some expression, we need to escape\nall the way out.\n\n\n\nWe could print a runtime error and then abort the process and exit the\napplication entirely. That has a certain melodramatic flair. Sort of the\nprogramming language interpreter equivalent of a mic drop.\n\nTempting as that is, we should probably do something a little less cataclysmic.\nWhile a runtime error needs to stop evaluating the *expression*, it shouldn't\nkill the *interpreter*. If a user is running the REPL and has a typo in a line\nof code, they should still be able to keep the session going and enter more code\nafter that.\n\n### Detecting runtime errors\n\nOur tree-walk interpreter evaluates nested expressions using recursive method\ncalls, and we need to unwind out of all of those. Throwing an exception in Java\nis a fine way to accomplish that. However, instead of using Java's own cast\nfailure, we'll define a Lox-specific one so that we can handle it how we want.\n\nBefore we do the cast, we check the object's type ourselves. So, for unary `-`,\nwe add:\n\n^code check-unary-operand (1 before, 1 after)\n\nThe code to check the operand is:\n\n^code check-operand\n\nWhen the check fails, it throws one of these:\n\n^code runtime-error-class\n\nUnlike the Java cast exception, our class tracks the\ntoken that identifies where in the user's code the runtime error came from. As\nwith static errors, this helps the user know where to fix their code.\n\n\n\nWe need similar checking for the binary operators. Since I promised you every\nsingle line of code needed to implement the interpreters, I'll run through them\nall.\n\nGreater than:\n\n^code check-greater-operand (1 before, 1 after)\n\nGreater than or equal to:\n\n^code check-greater-equal-operand (1 before, 1 after)\n\nLess than:\n\n^code check-less-operand (1 before, 1 after)\n\nLess than or equal to:\n\n^code check-less-equal-operand (1 before, 1 after)\n\nSubtraction:\n\n^code check-minus-operand (1 before, 1 after)\n\nDivision:\n\n^code check-slash-operand (1 before, 1 after)\n\nMultiplication:\n\n^code check-star-operand (1 before, 1 after)\n\nAll of those rely on this validator, which is virtually the same as the unary\none:\n\n^code check-operands\n\n\n\nThe last remaining operator, again the odd one out, is addition. Since `+` is\noverloaded for numbers and strings, it already has code to check the types. All\nwe need to do is fail if neither of the two success cases match.\n\n^code string-wrong-type (3 before, 1 after)\n\nThat gets us detecting runtime errors deep in the innards of the evaluator. The\nerrors are getting thrown. The next step is to write the code that catches them.\nFor that, we need to wire up the Interpreter class into the main Lox class that\ndrives it.\n\n## Hooking Up the Interpreter\n\nThe visit methods are sort of the guts of the Interpreter class, where the real\nwork happens. We need to wrap a skin around them to interface with the rest of\nthe program. The Interpreter's public API is simply one method.\n\n^code interpret\n\nThis takes in a syntax tree for an expression and evaluates it. If that\nsucceeds, `evaluate()` returns an object for the result value. `interpret()`\nconverts that to a string and shows it to the user. To convert a Lox value to a\nstring, we rely on:\n\n^code stringify\n\nThis is another of those pieces of code like `isTruthy()` that crosses the\nmembrane between the user's view of Lox objects and their internal\nrepresentation in Java.\n\nIt's pretty straightforward. Since Lox was designed to be familiar to someone\ncoming from Java, things like Booleans look the same in both languages. The two\nedge cases are `nil`, which we represent using Java's `null`, and numbers.\n\nLox uses double-precision numbers even for integer values. In that case, they\nshould print without a decimal point. Since Java has both floating point and\ninteger types, it wants you to know which one you're using. It tells you by\nadding an explicit `.0` to integer-valued doubles. We don't care about that, so\nwe hack it off the end.\n\n\n\n### Reporting runtime errors\n\nIf a runtime error is thrown while evaluating the expression, `interpret()`\ncatches it. This lets us report the error to the user and then gracefully\ncontinue. All of our existing error reporting code lives in the Lox class, so we\nput this method there too:\n\n^code runtime-error-method\n\nWe use the token associated with the RuntimeError to tell the user what line of\ncode was executing when the error occurred. Even better would be to give the\nuser an entire call stack to show how they *got* to be executing that code. But\nwe don't have function calls yet, so I guess we don't have to worry about it.\n\nAfter showing the error, `runtimeError()` sets this field:\n\n^code had-runtime-error-field (1 before, 1 after)\n\nThat field plays a small but important role.\n\n^code check-runtime-error (4 before, 1 after)\n\nIf the user is running a Lox script from a file and a\nruntime error occurs, we set an exit code when the process quits to let the\ncalling process know. Not everyone cares about shell etiquette, but we do.\n\n\n\n### Running the interpreter\n\nNow that we have an interpreter, the Lox class can start using it.\n\n^code interpreter-instance (1 before, 1 after)\n\nWe make the field static so that successive calls to `run()` inside a REPL\nsession reuse the same interpreter. That doesn't make a difference now, but it\nwill later when the interpreter stores global variables. Those variables should\npersist throughout the REPL session.\n\nFinally, we remove the line of temporary code from the [last chapter][] for\nprinting the syntax tree and replace it with this:\n\n[last chapter]: parsing-expressions.html\n\n^code interpreter-interpret (3 before, 1 after)\n\nWe have an entire language pipeline now: scanning, parsing, and\nexecution. Congratulations, you now have your very own arithmetic calculator.\n\nAs you can see, the interpreter is pretty bare bones. But the Interpreter class\nand the Visitor pattern we've set up today form the skeleton that later chapters\nwill stuff full of interesting guts -- variables, functions, etc. Right now, the\ninterpreter doesn't do very much, but it's alive!\n\n\"A\n\n
\n\n## Challenges\n\n1. Allowing comparisons on types other than numbers could be useful. The\n operators might have a reasonable interpretation for strings. Even\n comparisons among mixed types, like `3 < \"pancake\"` could be handy to enable\n things like ordered collections of heterogeneous types. Or it could simply\n lead to bugs and confusion.\n\n Would you extend Lox to support comparing other types? If so, which pairs of\n types do you allow and how do you define their ordering? Justify your\n choices and compare them to other languages.\n\n2. Many languages define `+` such that if *either* operand is a string, the\n other is converted to a string and the results are then concatenated. For\n example, `\"scone\" + 4` would yield `scone4`. Extend the code in\n `visitBinaryExpr()` to support that.\n\n3. What happens right now if you divide a number by zero? What do you think\n should happen? Justify your choice. How do other languages you know handle\n division by zero, and why do they make the choices they do?\n\n Change the implementation in `visitBinaryExpr()` to detect and report a\n runtime error for this case.\n\n
\n\n
\n\n## Design Note: Static and Dynamic Typing\n\nSome languages, like Java, are statically typed which means type errors are\ndetected and reported at compile time before any code is run. Others, like Lox,\nare dynamically typed and defer checking for type errors until runtime right\nbefore an operation is attempted. We tend to consider this a black-and-white\nchoice, but there is actually a continuum between them.\n\nIt turns out even most statically typed languages do *some* type checks at\nruntime. The type system checks most type rules statically, but inserts runtime\nchecks in the generated code for other operations.\n\nFor example, in Java, the *static* type system assumes a cast expression will\nalways safely succeed. After you cast some value, you can statically treat it as\nthe destination type and not get any compile errors. But downcasts can fail,\nobviously. The only reason the static checker can presume that casts always\nsucceed without violating the language's soundness guarantees, is because the\ncast is checked *at runtime* and throws an exception on failure.\n\nA more subtle example is [covariant arrays][] in Java and C#. The static\nsubtyping rules for arrays allow operations that are not sound. Consider:\n\n[covariant arrays]: https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science)#Covariant_arrays_in_Java_and_C.23\n\n```java\nObject[] stuff = new Integer[1];\nstuff[0] = \"not an int!\";\n```\n\nThis code compiles without any errors. The first line upcasts the Integer array\nand stores it in a variable of type Object array. The second line stores a\nstring in one of its cells. The Object array type statically allows that\n-- strings *are* Objects -- but the actual Integer array that `stuff` refers to\nat runtime should never have a string in it! To avoid that catastrophe, when you\nstore a value in an array, the JVM does a *runtime* check to make sure it's an\nallowed type. If not, it throws an ArrayStoreException.\n\nJava could have avoided the need to check this at runtime by disallowing the\ncast on the first line. It could make arrays *invariant* such that an array of\nIntegers is *not* an array of Objects. That's statically sound, but it prohibits\ncommon and safe patterns of code that only read from arrays. Covariance is safe\nif you never *write* to the array. Those patterns were particularly important\nfor usability in Java 1.0 before it supported generics. James Gosling and the\nother Java designers traded off a little static safety and performance -- those\narray store checks take time -- in return for some flexibility.\n\nThere are few modern statically typed languages that don't make that trade-off\n*somewhere*. Even Haskell will let you run code with non-exhaustive matches. If\nyou find yourself designing a statically typed language, keep in mind that you\ncan sometimes give users more flexibility without sacrificing *too* many of the\nbenefits of static safety by deferring some type checks until runtime.\n\nOn the other hand, a key reason users choose statically typed languages is\nbecause of the confidence the language gives them that certain kinds of errors\ncan *never* occur when their program is run. Defer too many type checks until\nruntime, and you erode that confidence.\n\n
\n" }, { "path": "book/functions.md", "content": "> And that is also the way the human mind works -- by the compounding of old\n> ideas into new structures that become new ideas that can themselves be used in\n> compounds, and round and round endlessly, growing ever more remote from the\n> basic earthbound imagery that is each language's soil.\n>\n> Douglas R. Hofstadter, I Am a Strange Loop\n\nThis chapter marks the culmination of a lot of hard work. The previous chapters\nadd useful functionality in their own right, but each also supplies a piece of a\npuzzle. We'll take those pieces -- expressions,\nstatements, variables, control flow, and lexical scope -- add a couple more, and\nassemble them all into support for real user-defined functions and function\ncalls.\n\n\n\n## Function Calls\n\nYou're certainly familiar with C-style function call syntax, but the grammar is\nmore subtle than you may realize. Calls are typically to named functions like:\n\n```lox\naverage(1, 2);\n```\n\nBut the name of the function being called isn't\nactually part of the call syntax. The thing being called -- the **callee** --\ncan be any expression that evaluates to a function. (Well, it does have to be a\npretty *high precedence* expression, but parentheses take care of that.) For\nexample:\n\n\n\n```lox\ngetCallback()();\n```\n\nThere are two call expressions here. The first pair of parentheses has\n`getCallback` as its callee. But the second call has the entire `getCallback()`\nexpression as its callee. It is the parentheses following an expression that\nindicate a function call. You can think of a call as sort of like a postfix\noperator that starts with `(`.\n\nThis \"operator\" has higher precedence than any other operator, even the unary\nones. So we slot it into the grammar by having the `unary` rule bubble up to a\nnew `call` rule.\n\n\n\n```ebnf\nunary → ( \"!\" | \"-\" ) unary | call ;\ncall → primary ( \"(\" arguments? \")\" )* ;\n```\n\nThis rule matches a primary expression followed by zero or more function calls.\nIf there are no parentheses, this parses a bare primary expression. Otherwise,\neach call is recognized by a pair of parentheses with an optional list of\narguments inside. The argument list grammar is:\n\n\n\n```ebnf\narguments → expression ( \",\" expression )* ;\n```\n\nThis rule requires at least one argument expression, followed by zero or more\nother expressions, each preceded by a comma. To handle zero-argument calls, the\n`call` rule itself considers the entire `arguments` production to be optional.\n\nI admit, this seems more grammatically awkward than you'd expect for the\nincredibly common \"zero or more comma-separated things\" pattern. There are some\nsophisticated metasyntaxes that handle this better, but in our BNF and in many\nlanguage specs I've seen, it is this cumbersome.\n\nOver in our syntax tree generator, we add a new\nnode.\n\n^code call-expr (1 before, 1 after)\n\n\n\nIt stores the callee expression and a list of expressions for the arguments. It\nalso stores the token for the closing parenthesis. We'll use that token's\nlocation when we report a runtime error caused by a function call.\n\nCrack open the parser. Where `unary()` used to jump straight to `primary()`,\nchange it to call, well, `call()`.\n\n^code unary-call (3 before, 1 after)\n\nIts definition is:\n\n^code call\n\nThe code here doesn't quite line up with the grammar rules. I moved a few things\naround to make the code cleaner -- one of the luxuries we have with a\nhandwritten parser. But it's roughly similar to how we parse infix operators.\nFirst, we parse a primary expression, the \"left operand\" to the call. Then, each\ntime we see a `(`, we call `finishCall()` to parse the call expression using the\npreviously parsed expression as the callee. The returned expression becomes the\nnew `expr` and we loop to see if the result is itself called.\n\n\n\nThe code to parse the argument list is in this helper:\n\n^code finish-call\n\nThis is more or less the `arguments` grammar rule translated to code, except\nthat we also handle the zero-argument case. We check for that case first by\nseeing if the next token is `)`. If it is, we don't try to parse any arguments.\n\nOtherwise, we parse an expression, then look for a comma indicating that there\nis another argument after that. We keep doing that as long as we find commas\nafter each expression. When we don't find a comma, then the argument list must\nbe done and we consume the expected closing parenthesis. Finally, we wrap the\ncallee and those arguments up into a call AST node.\n\n### Maximum argument counts\n\nRight now, the loop where we parse arguments has no bound. If you want to call a\nfunction and pass a million arguments to it, the parser would have no problem\nwith it. Do we want to limit that?\n\nOther languages have various approaches. The C standard says a conforming\nimplementation has to support *at least* 127 arguments to a function, but\ndoesn't say there's any upper limit. The Java specification says a method can\naccept *no more than* 255 arguments.\n\n\n\nOur Java interpreter for Lox doesn't really need a limit, but having a maximum\nnumber of arguments will simplify our bytecode interpreter in [Part III][]. We\nwant our two interpreters to be compatible with each other, even in weird corner\ncases like this, so we'll add the same limit to jlox.\n\n[part iii]: a-bytecode-virtual-machine.html\n\n^code check-max-arity (1 before, 1 after)\n\nNote that the code here *reports* an error if it encounters too many arguments,\nbut it doesn't *throw* the error. Throwing is how we kick into panic mode which\nis what we want if the parser is in a confused state and doesn't know where it\nis in the grammar anymore. But here, the parser is still in a perfectly valid\nstate -- it just found too many arguments. So it reports the error and keeps on\nkeepin' on.\n\n### Interpreting function calls\n\nWe don't have any functions we can call, so it seems weird to start implementing\ncalls first, but we'll worry about that when we get there. First, our\ninterpreter needs a new import.\n\n^code import-array-list (1 after)\n\nAs always, interpretation starts with a new visit method for our new call\nexpression node.\n\n^code visit-call\n\nFirst, we evaluate the expression for the callee. Typically, this expression is\njust an identifier that looks up the function by its name, but it could be\nanything. Then we evaluate each of the argument expressions in order and store\nthe resulting values in a list.\n\n\n\nOnce we've got the callee and the arguments ready, all that remains is to\nperform the call. We do that by casting the callee to a LoxCallable and then invoking a `call()` method on it.\nThe Java representation of any Lox object that can be called like a function\nwill implement this interface. That includes user-defined functions, naturally,\nbut also class objects since classes are \"called\" to construct new instances.\nWe'll also use it for one more purpose shortly.\n\n\n\nThere isn't too much to this new interface.\n\n^code callable\n\nWe pass in the interpreter in case the class implementing `call()` needs it. We\nalso give it the list of evaluated argument values. The implementer's job is\nthen to return the value that the call expression produces.\n\n### Call type errors\n\nBefore we get to implementing LoxCallable, we need to make the visit method a\nlittle more robust. It currently ignores a couple of failure modes that we can't\npretend won't occur. First, what happens if the callee isn't actually something\nyou can call? What if you try to do this:\n\n```lox\n\"totally not a function\"();\n```\n\nStrings aren't callable in Lox. The runtime representation of a Lox string is a\nJava string, so when we cast that to LoxCallable, the JVM will throw a\nClassCastException. We don't want our interpreter to vomit out some nasty Java\nstack trace and die. Instead, we need to check the type ourselves first.\n\n^code check-is-callable (2 before, 1 after)\n\nWe still throw an exception, but now we're throwing our own exception type, one\nthat the interpreter knows to catch and report gracefully.\n\n### Checking arity\n\nThe other problem relates to the function's **arity**. Arity is the fancy term\nfor the number of arguments a function or operation expects. Unary operators\nhave arity one, binary operators two, etc. With functions, the arity is\ndetermined by the number of parameters it declares.\n\n```lox\nfun add(a, b, c) {\n print a + b + c;\n}\n```\n\nThis function defines three parameters, `a`, `b`, and `c`, so its arity is\nthree and it expects three arguments. So what if you try to call it like this:\n\n```lox\nadd(1, 2, 3, 4); // Too many.\nadd(1, 2); // Too few.\n```\n\nDifferent languages take different approaches to this problem. Of course, most\nstatically typed languages check this at compile time and refuse to compile the\ncode if the argument count doesn't match the function's arity. JavaScript\ndiscards any extra arguments you pass. If you don't pass enough, it fills in the\nmissing parameters with the magic sort-of-like-null-but-not-really value\n`undefined`. Python is stricter. It raises a runtime error if the argument list\nis too short or too long.\n\nI think the latter is a better approach. Passing the wrong number of arguments\nis almost always a bug, and it's a mistake I do make in practice. Given that,\nthe sooner the implementation draws my attention to it, the better. So for Lox,\nwe'll take Python's approach. Before invoking the callable, we check to see if\nthe argument list's length matches the callable's arity.\n\n^code check-arity (1 before, 1 after)\n\nThat requires a new method on the LoxCallable interface to ask it its arity.\n\n^code callable-arity (1 before, 1 after)\n\nWe *could* push the arity checking into the concrete implementation of `call()`.\nBut, since we'll have multiple classes implementing LoxCallable, that would end\nup with redundant validation spread across a few classes. Hoisting it up into\nthe visit method lets us do it in one place.\n\n## Native Functions\n\nWe can theoretically call functions, but we have no functions to call yet.\nBefore we get to user-defined functions, now is a good time to introduce a vital\nbut often overlooked facet of language implementations -- **native functions**. These are functions that the\ninterpreter exposes to user code but that are implemented in the host language\n(in our case Java), not the language being implemented (Lox).\n\nSometimes these are called **primitives**, **external functions**, or **foreign\nfunctions**. Since these functions can be called while the user's program is\nrunning, they form part of the implementation's runtime. A lot of programming\nlanguage books gloss over these because they aren't conceptually interesting.\nThey're mostly grunt work.\n\n\n\nBut when it comes to making your language actually good at doing useful stuff,\nthe native functions your implementation provides are key. They provide access\nto the fundamental services that all programs are defined in terms of. If you\ndon't provide native functions to access the file system, a user's going to have\na hell of a time writing a program that reads and displays a file.\n\n\n\nMany languages also allow users to provide their own native functions. The\nmechanism for doing so is called a **foreign function interface** (**FFI**),\n**native extension**, **native interface**, or something along those lines.\nThese are nice because they free the language implementer from providing access\nto every single capability the underlying platform supports. We won't define an\nFFI for jlox, but we will add one native function to give you an idea of what it\nlooks like.\n\n### Telling time\n\nWhen we get to [Part III][] and start working on a much more efficient\nimplementation of Lox, we're going to care deeply about performance. Performance\nwork requires measurement, and that in turn means **benchmarks**. These are\nprograms that measure the time it takes to exercise some corner of the\ninterpreter.\n\nWe could measure the time it takes to start up the interpreter, run the\nbenchmark, and exit, but that adds a lot of overhead -- JVM startup time, OS\nshenanigans, etc. That stuff does matter, of course, but if you're just trying\nto validate an optimization to some piece of the interpreter, you don't want\nthat overhead obscuring your results.\n\nA nicer solution is to have the benchmark script itself measure the time elapsed\nbetween two points in the code. To do that, a Lox program needs to be able to\ntell time. There's no way to do that now -- you can't implement a useful clock\n\"from scratch\" without access to the underlying clock on the computer.\n\nSo we'll add `clock()`, a native function that returns the number of seconds\nthat have passed since some fixed point in time. The difference between two\nsuccessive invocations tells you how much time elapsed between the two calls.\nThis function is defined in the global scope, so let's ensure the interpreter\nhas access to that.\n\n^code global-environment (2 before, 2 after)\n\nThe `environment` field in the interpreter changes as we enter and exit local\nscopes. It tracks the *current* environment. This new `globals` field holds a\nfixed reference to the outermost global environment.\n\nWhen we instantiate an Interpreter, we stuff the native function in that global\nscope.\n\n^code interpreter-constructor (2 before, 1 after)\n\nThis defines a variable named \"clock\". Its value is a\nJava anonymous class that implements LoxCallable. The `clock()` function takes\nno arguments, so its arity is zero. The implementation of `call()` calls the\ncorresponding Java function and converts the result to a double value in\nseconds.\n\n\n\nIf we wanted to add other native functions -- reading input from the user,\nworking with files, etc. -- we could add them each as their own anonymous class\nthat implements LoxCallable. But for the book, this one is really all we need.\n\nLet's get ourselves out of the function-defining business and let our users\ntake over...\n\n## Function Declarations\n\nWe finally get to add a new production to the `declaration` rule we introduced\nback when we added variables. Function declarations, like variables, bind a new\nname. That means they are allowed only in places where\na declaration is permitted.\n\n\n\n```ebnf\ndeclaration → funDecl\n | varDecl\n | statement ;\n```\n\nThe updated `declaration` rule references this new rule:\n\n```ebnf\nfunDecl → \"fun\" function ;\nfunction → IDENTIFIER \"(\" parameters? \")\" block ;\n```\n\nThe main `funDecl` rule uses a separate helper rule `function`. A function\n*declaration statement* is the `fun` keyword followed by the actual function-y\nstuff. When we get to classes, we'll reuse that `function` rule for declaring\nmethods. Those look similar to function declarations, but aren't preceded by\n`fun`.\n\n\n\nThe function itself is a name followed by the parenthesized parameter list and\nthe body. The body is always a braced block, using the same grammar rule that\nblock statements use. The parameter list uses this rule:\n\n```ebnf\nparameters → IDENTIFIER ( \",\" IDENTIFIER )* ;\n```\n\nIt's like the earlier `arguments` rule, except that each parameter is an\nidentifier, not an expression. That's a lot of new syntax for the parser to chew\nthrough, but the resulting AST node isn't too bad.\n\n^code function-ast (1 before, 1 after)\n\n\n\nA function node has a name, a list of parameters (their names), and then the\nbody. We store the body as the list of statements contained inside the curly\nbraces.\n\nOver in the parser, we weave in the new declaration.\n\n^code match-fun (1 before, 1 after)\n\nLike other statements, a function is recognized by the leading keyword. When we\nencounter `fun`, we call `function`. That corresponds to the `function` grammar\nrule since we already matched and consumed the `fun` keyword. We'll build the\nmethod up a piece at a time, starting with this:\n\n^code parse-function\n\nRight now, it only consumes the identifier token for the function's name. You\nmight be wondering about that funny little `kind` parameter. Just like we reuse\nthe grammar rule, we'll reuse the `function()` method later to parse methods\ninside classes. When we do that, we'll pass in \"method\" for `kind` so that the\nerror messages are specific to the kind of declaration being parsed.\n\nNext, we parse the parameter list and the pair of parentheses wrapped around it.\n\n^code parse-parameters (1 before, 1 after)\n\nThis is like the code for handling arguments in a call, except not split out\ninto a helper method. The outer `if` statement handles the zero parameter case,\nand the inner `while` loop parses parameters as long as we find commas to\nseparate them. The result is the list of tokens for each parameter's name.\n\nJust like we do with arguments to function calls, we validate at parse time\nthat you don't exceed the maximum number of parameters a function is allowed to\nhave.\n\nFinally, we parse the body and wrap it all up in a function node.\n\n^code parse-body (1 before, 1 after)\n\nNote that we consume the `{` at the beginning of the body here before calling\n`block()`. That's because `block()` assumes the brace token has already been\nmatched. Consuming it here lets us report a more precise error message if the\n`{` isn't found since we know it's in the context of a function declaration.\n\n## Function Objects\n\nWe've got some syntax parsed so usually we're ready to interpret, but first we\nneed to think about how to represent a Lox function in Java. We need to keep\ntrack of the parameters so that we can bind them to argument values when the\nfunction is called. And, of course, we need to keep the code for the body of the\nfunction so that we can execute it.\n\nThat's basically what the Stmt.Function class is. Could we just use that?\nAlmost, but not quite. We also need a class that implements LoxCallable so that\nwe can call it. We don't want the runtime phase of the interpreter to bleed into\nthe front end's syntax classes so we don't want Stmt.Function itself to\nimplement that. Instead, we wrap it in a new class.\n\n^code lox-function\n\nWe implement the `call()` of LoxCallable like so:\n\n^code function-call\n\nThis handful of lines of code is one of the most fundamental, powerful pieces of\nour interpreter. As we saw in [the chapter on statements and state][statements], managing name environments is a core part\nof a language implementation. Functions are deeply tied to that.\n\n[statements]: statements-and-state.html\n\n\n\nParameters are core to functions, especially the fact that a function\n*encapsulates* its parameters -- no other code outside of the function can see\nthem. This means each function gets its own environment where it stores those\nvariables.\n\nFurther, this environment must be created dynamically. Each function *call* gets\nits own environment. Otherwise, recursion would break. If there are multiple\ncalls to the same function in play at the same time, each needs its *own*\nenvironment, even though they are all calls to the same function.\n\nFor example, here's a convoluted way to count to three:\n\n```lox\nfun count(n) {\n if (n > 1) count(n - 1);\n print n;\n}\n\ncount(3);\n```\n\nImagine we pause the interpreter right at the point where it's about to print 1\nin the innermost nested call. The outer calls to print 2 and 3 haven't printed\ntheir values yet, so there must be environments somewhere in memory that still\nstore the fact that `n` is bound to 3 in one context, 2 in another, and 1 in the\ninnermost, like:\n\n\"A\n\nThat's why we create a new environment at each *call*, not at the function\n*declaration*. The `call()` method we saw earlier does that. At the beginning of\nthe call, it creates a new environment. Then it walks the parameter and argument\nlists in lockstep. For each pair, it creates a new variable with the parameter's\nname and binds it to the argument's value.\n\nSo, for a program like this:\n\n```lox\nfun add(a, b, c) {\n print a + b + c;\n}\n\nadd(1, 2, 3);\n```\n\nAt the point of the call to `add()`, the interpreter creates something like\nthis:\n\n\"Binding\n\nThen `call()` tells the interpreter to execute the body of the function in this\nnew function-local environment. Up until now, the current environment was the\nenvironment where the function was being called. Now, we teleport from there\ninside the new parameter space we've created for the function.\n\nThis is all that's required to pass data into the function. By using different\nenvironments when we execute the body, calls to the same function with the\nsame code can produce different results.\n\nOnce the body of the function has finished executing, `executeBlock()` discards\nthat function-local environment and restores the previous one that was active\nback at the callsite. Finally, `call()` returns `null`, which returns `nil` to\nthe caller. (We'll add return values later.)\n\nMechanically, the code is pretty simple. Walk a couple of lists. Bind some new\nvariables. Call a method. But this is where the crystalline *code* of the\nfunction declaration becomes a living, breathing *invocation*. This is one of my\nfavorite snippets in this entire book. Feel free to take a moment to meditate on\nit if you're so inclined.\n\nDone? OK. Note when we bind the parameters, we assume the parameter and argument\nlists have the same length. This is safe because `visitCallExpr()` checks the\narity before calling `call()`. It relies on the function reporting its arity to\ndo that.\n\n^code function-arity\n\nThat's most of our object representation. While we're in here, we may as well\nimplement `toString()`.\n\n^code function-to-string\n\nThis gives nicer output if a user decides to print a function value.\n\n```lox\nfun add(a, b) {\n print a + b;\n}\n\nprint add; // \"\".\n```\n\n### Interpreting function declarations\n\nWe'll come back and refine LoxFunction soon, but that's enough to get started.\nNow we can visit a function declaration.\n\n^code visit-function\n\nThis is similar to how we interpret other literal expressions. We take a\nfunction *syntax node* -- a compile-time representation of the function -- and\nconvert it to its runtime representation. Here, that's a LoxFunction that wraps\nthe syntax node.\n\nFunction declarations are different from other literal nodes in that the\ndeclaration *also* binds the resulting object to a new variable. So, after\ncreating the LoxFunction, we create a new binding in the current environment and\nstore a reference to it there.\n\nWith that, we can define and call our own functions all within Lox. Give it a\ntry:\n\n```lox\nfun sayHi(first, last) {\n print \"Hi, \" + first + \" \" + last + \"!\";\n}\n\nsayHi(\"Dear\", \"Reader\");\n```\n\nI don't know about you, but that looks like an honest-to-God programming\nlanguage to me.\n\n## Return Statements\n\nWe can get data into functions by passing parameters, but we've got no way to\nget results back *out*. If Lox were an\nexpression-oriented language like Ruby or Scheme, the body would be an\nexpression whose value is implicitly the function's result. But in Lox, the body\nof a function is a list of statements which don't produce values, so we need\ndedicated syntax for emitting a result. In other words, `return` statements. I'm\nsure you can guess the grammar already.\n\n\n\n```ebnf\nstatement → exprStmt\n | forStmt\n | ifStmt\n | printStmt\n | returnStmt\n | whileStmt\n | block ;\n\nreturnStmt → \"return\" expression? \";\" ;\n```\n\nWe've got one more -- the final, in fact -- production under the venerable\n`statement` rule. A `return` statement is the `return` keyword followed by an\noptional expression and terminated with a semicolon.\n\nThe return value is optional to support exiting early from a function that\ndoesn't return a useful value. In statically typed languages, \"void\" functions\ndon't return a value and non-void ones do. Since Lox is dynamically typed, there\nare no true void functions. The compiler has no way of preventing you from\ntaking the result value of a call to a function that doesn't contain a `return`\nstatement.\n\n```lox\nfun procedure() {\n print \"don't return anything\";\n}\n\nvar result = procedure();\nprint result; // ?\n```\n\nThis means every Lox function must return *something*, even if it contains no\n`return` statements at all. We use `nil` for this, which is why LoxFunction's\nimplementation of `call()` returns `null` at the end. In that same vein, if you\nomit the value in a `return` statement, we simply treat it as equivalent to:\n\n```lox\nreturn nil;\n```\n\nOver in our AST generator, we add a new node.\n\n^code return-ast (1 before, 1 after)\n\n\n\nIt keeps the `return` keyword token so we can use its location for error\nreporting, and the value being returned, if any. We parse it like other\nstatements, first by recognizing the initial keyword.\n\n^code match-return (1 before, 1 after)\n\nThat branches out to:\n\n^code parse-return-statement\n\nAfter snagging the previously consumed `return` keyword, we look for a value\nexpression. Since many different tokens can potentially start an expression,\nit's hard to tell if a return value is *present*. Instead, we check if it's\n*absent*. Since a semicolon can't begin an expression, if the next token is\nthat, we know there must not be a value.\n\n### Returning from calls\n\nInterpreting a `return` statement is tricky. You can return from anywhere within\nthe body of a function, even deeply nested inside other statements. When the\nreturn is executed, the interpreter needs to jump all the way out of whatever\ncontext it's currently in and cause the function call to complete, like some\nkind of jacked up control flow construct.\n\nFor example, say we're running this program and we're about to execute the\n`return` statement:\n\n```lox\nfun count(n) {\n while (n < 100) {\n if (n == 3) return n; // <--\n print n;\n n = n + 1;\n }\n}\n\ncount(1);\n```\n\nThe Java call stack currently looks roughly like this:\n\n```text\nInterpreter.visitReturnStmt()\nInterpreter.visitIfStmt()\nInterpreter.executeBlock()\nInterpreter.visitBlockStmt()\nInterpreter.visitWhileStmt()\nInterpreter.executeBlock()\nLoxFunction.call()\nInterpreter.visitCallExpr()\n```\n\nWe need to get from the top of the stack all the way back to `call()`. I don't\nknow about you, but to me that sounds like exceptions. When we execute a\n`return` statement, we'll use an exception to unwind the interpreter past the\nvisit methods of all of the containing statements back to the code that began\nexecuting the body.\n\nThe visit method for our new AST node looks like this:\n\n^code visit-return\n\nIf we have a return value, we evaluate it, otherwise, we use `nil`. Then we take\nthat value and wrap it in a custom exception class and throw it.\n\n^code return-exception\n\nThis class wraps the return value with the accoutrements Java requires for a\nruntime exception class. The weird super constructor call with those `null` and\n`false` arguments disables some JVM machinery that we don't need. Since we're\nusing our exception class for control flow and not\nactual error handling, we don't need overhead like stack traces.\n\n\n\nWe want this to unwind all the way to where the function call began, the\n`call()` method in LoxFunction.\n\n^code catch-return (3 before, 1 after)\n\nWe wrap the call to `executeBlock()` in a try-catch block. When it catches a\nreturn exception, it pulls out the value and makes that the return value from\n`call()`. If it never catches one of these exceptions, it means the function\nreached the end of its body without hitting a `return` statement. In that case,\nit implicitly returns `nil`.\n\nLet's try it out. We finally have enough power to support this classic\nexample -- a recursive function to calculate Fibonacci numbers:\n\n\n\n```lox\nfun fib(n) {\n if (n <= 1) return n;\n return fib(n - 2) + fib(n - 1);\n}\n\nfor (var i = 0; i < 20; i = i + 1) {\n print fib(i);\n}\n```\n\nThis tiny program exercises almost every language feature we have spent the past\nseveral chapters implementing -- expressions, arithmetic, branching, looping,\nvariables, functions, function calls, parameter binding, and returns.\n\n\n\n## Local Functions and Closures\n\nOur functions are pretty full featured, but there is one hole to patch. In fact,\nit's a big enough gap that we'll spend most of the [next chapter][] sealing it\nup, but we can get started here.\n\nLoxFunction's implementation of `call()` creates a new environment where it\nbinds the function's parameters. When I showed you that code, I glossed over one\nimportant point: What is the *parent* of that environment?\n\nRight now, it is always `globals`, the top-level global environment. That way,\nif an identifier isn't defined inside the function body itself, the interpreter\ncan look outside the function in the global scope to find it. In the Fibonacci\nexample, that's how the interpreter is able to look up the recursive call to\n`fib` inside the function's own body -- `fib` is a global variable.\n\nBut recall that in Lox, function declarations are allowed *anywhere* a name can\nbe bound. That includes the top level of a Lox script, but also the inside of\nblocks or other functions. Lox supports **local functions** that are defined\ninside another function, or nested inside a block.\n\nConsider this classic example:\n\n```lox\nfun makeCounter() {\n var i = 0;\n fun count() {\n i = i + 1;\n print i;\n }\n\n return count;\n}\n\nvar counter = makeCounter();\ncounter(); // \"1\".\ncounter(); // \"2\".\n```\n\nHere, `count()` uses `i`, which is declared outside of itself in the containing\nfunction `makeCounter()`. `makeCounter()` returns a reference to the `count()`\nfunction and then its own body finishes executing completely.\n\nMeanwhile, the top-level code invokes the returned `count()` function. That\nexecutes the body of `count()`, which assigns to and reads `i`, even though the\nfunction where `i` was defined has already exited.\n\nIf you've never encountered a language with nested functions before, this might\nseem crazy, but users do expect it to work. Alas, if you run it now, you get an\nundefined variable error in the call to `counter()` when the body of `count()`\ntries to look up `i`. That's because the environment chain in effect looks like\nthis:\n\n\"The\n\nWhen we call `count()` (through the reference to it stored in `counter`), we\ncreate a new empty environment for the function body. The parent of that is the\nglobal environment. We lost the environment for `makeCounter()` where `i` is\nbound.\n\nLet's go back in time a bit. Here's what the environment chain looked like right\nwhen we declared `count()` inside the body of `makeCounter()`:\n\n\"The\n\nSo at the point where the function is declared, we can see `i`. But when we\nreturn from `makeCounter()` and exit its body, the interpreter discards that\nenvironment. Since the interpreter doesn't keep the environment surrounding\n`count()` around, it's up to the function object itself to hang on to it.\n\nThis data structure is called a **closure** because\nit \"closes over\" and holds on to the surrounding variables where the function is\ndeclared. Closures have been around since the early Lisp days, and language\nhackers have come up with all manner of ways to implement them. For jlox, we'll\ndo the simplest thing that works. In LoxFunction, we add a field to store an\nenvironment.\n\n\n\n^code closure-field (1 before, 1 after)\n\nWe initialize that in the constructor.\n\n^code closure-constructor (1 after)\n\nWhen we create a LoxFunction, we capture the current environment.\n\n^code visit-closure (1 before, 1 after)\n\nThis is the environment that is active when the function is *declared* not when\nit's *called*, which is what we want. It represents the lexical scope\nsurrounding the function declaration. Finally, when we call the function, we use\nthat environment as the call's parent instead of going straight to `globals`.\n\n^code call-closure (1 before, 1 after)\n\nThis creates an environment chain that goes from the function's body out through\nthe environments where the function is declared, all the way out to the global\nscope. The runtime environment chain matches the textual nesting of the source\ncode like we want. The end result when we call that function looks like this:\n\n\"The\n\nNow, as you can see, the interpreter can still find `i` when it needs to because\nit's in the middle of the environment chain. Try running that `makeCounter()`\nexample now. It works!\n\nFunctions let us abstract over, reuse, and compose code. Lox is much more\npowerful than the rudimentary arithmetic calculator it used to be. Alas, in our\nrush to cram closures in, we have let a tiny bit of dynamic scoping leak into\nthe interpreter. In the [next chapter][], we will explore deeper into lexical\nscope and close that hole.\n\n[next chapter]: resolving-and-binding.html\n\n
\n\n## Challenges\n\n1. Our interpreter carefully checks that the number of arguments passed to a\n function matches the number of parameters it expects. Since this check is\n done at runtime on every call, it has a performance cost. Smalltalk\n implementations don't have that problem. Why not?\n\n1. Lox's function declaration syntax performs two independent operations. It\n creates a function and also binds it to a name. This improves usability for\n the common case where you do want to associate a name with the function.\n But in functional-styled code, you often want to create a function to\n immediately pass it to some other function or return it. In that case, it\n doesn't need a name.\n\n Languages that encourage a functional style usually support **anonymous\n functions** or **lambdas** -- an expression syntax that creates a function\n without binding it to a name. Add anonymous function syntax to Lox so that\n this works:\n\n ```lox\n fun thrice(fn) {\n for (var i = 1; i <= 3; i = i + 1) {\n fn(i);\n }\n }\n\n thrice(fun (a) {\n print a;\n });\n // \"1\".\n // \"2\".\n // \"3\".\n ```\n\n How do you handle the tricky case of an anonymous function expression\n occurring in an expression statement:\n\n ```lox\n fun () {};\n ```\n\n1. Is this program valid?\n\n ```lox\n fun scope(a) {\n var a = \"local\";\n }\n ```\n\n In other words, are a function's parameters in the *same* scope as its local\n variables, or in an outer scope? What does Lox do? What about other\n languages you are familiar with? What do you think a language *should* do?\n\n
\n" }, { "path": "book/garbage-collection.md", "content": "> I wanna, I wanna,
\n> I wanna, I wanna,
\n> I wanna be trash.
\n>\n> The Whip, “Trash”\n\nWe say Lox is a \"high-level\" language because it frees programmers from worrying\nabout details irrelevant to the problem they're solving. The user becomes an\nexecutive, giving the machine abstract goals and letting the lowly computer\nfigure out how to get there.\n\nDynamic memory allocation is a perfect candidate for automation. It's necessary\nfor a working program, tedious to do by hand, and yet still error-prone. The\ninevitable mistakes can be catastrophic, leading to crashes, memory corruption,\nor security violations. It's the kind of risky-yet-boring work that machines\nexcel at over humans.\n\nThis is why Lox is a **managed language**, which means that the language\nimplementation manages memory allocation and freeing on the user's behalf. When\na user performs an operation that requires some dynamic memory, the VM\nautomatically allocates it. The programmer never worries about deallocating\nanything. The machine ensures any memory the program is using sticks around as\nlong as needed.\n\nLox provides the illusion that the computer has an infinite amount of memory.\nUsers can allocate and allocate and allocate and never once think about where\nall these bytes are coming from. Of course, computers do not yet *have* infinite\nmemory. So the way managed languages maintain this illusion is by going behind\nthe programmer's back and reclaiming memory that the program no longer needs.\nThe component that does this is called a **garbage collector**.\n\n\n\n## Reachability\n\nThis raises a surprisingly difficult question: how does a VM tell what memory is\n*not* needed? Memory is only needed if it is read in the future, but short of\nhaving a time machine, how can an implementation tell what code the program\n*will* execute and which data it *will* use? Spoiler alert: VMs cannot travel\ninto the future. Instead, the language makes a conservative approximation: it considers a piece of\nmemory to still be in use if it *could possibly* be read in the future.\n\n\n\nThat sounds *too* conservative. Couldn't *any* bit of memory potentially be\nread? Actually, no, at least not in a memory-safe language like Lox. Here's an\nexample:\n\n```lox\nvar a = \"first value\";\na = \"updated\";\n// GC here.\nprint a;\n```\n\nSay we run the GC after the assignment has completed on the second line. The\nstring \"first value\" is still sitting in memory, but there is no way for the\nuser's program to ever get to it. Once `a` got reassigned, the program lost any\nreference to that string. We can safely free it. A value is **reachable** if\nthere is some way for a user program to reference it. Otherwise, like the string\n\"first value\" here, it is **unreachable**.\n\nMany values can be directly accessed by the VM. Take a look at:\n\n```lox\nvar global = \"string\";\n{\n var local = \"another\";\n print global + local;\n}\n```\n\nPause the program right after the two strings have been concatenated but before\nthe `print` statement has executed. The VM can reach `\"string\"` by looking\nthrough the global variable table and finding the entry for `global`. It can\nfind `\"another\"` by walking the value stack and hitting the slot for the local\nvariable `local`. It can even find the concatenated string `\"stringanother\"`\nsince that temporary value is also sitting on the VM's stack at the point when\nwe paused our program.\n\nAll of these values are called **roots**. A root is any object that the VM can\nreach directly without going through a reference in some other object. Most\nroots are global variables or on the stack, but as we'll see, there are a couple\nof other places the VM stores references to objects that it can find.\n\nOther values can be found by going through a reference inside another value.\nFields on instances of classes are the most obvious\ncase, but we don't have those yet. Even without those, our VM still has indirect\nreferences. Consider:\n\n\n\n```lox\nfun makeClosure() {\n var a = \"data\";\n\n fun f() { print a; }\n return f;\n}\n\n{\n var closure = makeClosure();\n // GC here.\n closure();\n}\n```\n\nSay we pause the program on the marked line and run the garbage collector. When\nthe collector is done and the program resumes, it will call the closure, which\nwill in turn print `\"data\"`. So the collector needs to *not* free that string.\nBut here's what the stack looks like when we pause the program:\n\n\"The\n\nThe `\"data\"` string is nowhere on it. It has already been hoisted off the stack\nand moved into the closed upvalue that the closure uses. The closure itself is\non the stack. But to get to the string, we need to trace through the closure and\nits upvalue array. Since it *is* possible for the user's program to do that, all\nof these indirectly accessible objects are also considered reachable.\n\n\"All\n\nThis gives us an inductive definition of reachability:\n\n* All roots are reachable.\n\n* Any object referred to from a reachable object is itself reachable.\n\nThese are the values that are still \"live\" and need to stay in memory. Any value\nthat *doesn't* meet this definition is fair game for the collector to reap.\nThat recursive pair of rules hints at a recursive algorithm we can use to free\nup unneeded memory:\n\n1. Starting with the roots, traverse through object references to find the\n full set of reachable objects.\n\n2. Free all objects *not* in that set.\n\nMany different garbage collection algorithms are in\nuse today, but they all roughly follow that same structure. Some may interleave\nthe steps or mix them, but the two fundamental operations are there. They mostly\ndiffer in *how* they perform each step.\n\n\n\n## Mark-Sweep Garbage Collection\n\nThe first managed language was Lisp, the second \"high-level\" language to be\ninvented, right after Fortran. John McCarthy considered using manual memory\nmanagement or reference counting, but eventually settled on (and coined) garbage\ncollection -- once the program was out of memory, it would go back and find\nunused storage it could reclaim.\n\n\n\nHe designed the very first, simplest garbage collection algorithm, called\n**mark-and-sweep** or just **mark-sweep**. Its description fits in three short\nparagraphs in the initial paper on Lisp. Despite its age and simplicity, the\nsame fundamental algorithm underlies many modern memory managers. Some corners\nof CS seem to be timeless.\n\nAs the name implies, mark-sweep works in two phases:\n\n* **Marking:** We start with the roots and traverse or *trace* through all of the objects those roots refer to.\n This is a classic graph traversal of all of the reachable objects. Each time\n we visit an object, we *mark* it in some way. (Implementations differ in how\n they record the mark.)\n\n* **Sweeping:** Once the mark phase completes, every reachable object\n in the heap has been marked. That means any unmarked object is unreachable and\n ripe for reclamation. We go through all the unmarked objects and free each\n one.\n\nIt looks something like this:\n\n\"Starting\n\n\n\nThat's what we're gonna implement. Whenever we decide it's time to reclaim some\nbytes, we'll trace everything and mark all the reachable objects, free what\ndidn't get marked, and then resume the user's program.\n\n### Collecting garbage\n\nThis entire chapter is about implementing this one function:\n\n\n\n^code collect-garbage-h (1 before, 1 after)\n\nWe'll work our way up to a full implementation starting with this empty shell:\n\n^code collect-garbage\n\nThe first question you might ask is, When does this function get called? It\nturns out that's a subtle question that we'll spend some time on later in the\nchapter. For now we'll sidestep the issue and build ourselves a handy diagnostic\ntool in the process.\n\n^code define-stress-gc (1 before, 2 after)\n\nWe'll add an optional \"stress test\" mode for the garbage collector. When this\nflag is defined, the GC runs as often as it possibly can. This is, obviously,\nhorrendous for performance. But it's great for flushing out memory management\nbugs that occur only when a GC is triggered at just the right moment. If *every*\nmoment triggers a GC, you're likely to find those bugs.\n\n^code call-collect (1 before, 1 after)\n\nWhenever we call `reallocate()` to acquire more memory, we force a collection to\nrun. The if check is because `reallocate()` is also called to free or shrink an\nallocation. We don't want to trigger a GC for that -- in particular because the\nGC itself will call `reallocate()` to free memory.\n\nCollecting right before allocation is the classic way\nto wire a GC into a VM. You're already calling into the memory manager, so it's\nan easy place to hook in the code. Also, allocation is the only time when you\nreally *need* some freed up memory so that you can reuse it. If you *don't* use\nallocation to trigger a GC, you have to make sure every possible place in code\nwhere you can loop and allocate memory also has a way to trigger the collector.\nOtherwise, the VM can get into a starved state where it needs more memory but\nnever collects any.\n\n\n\n### Debug logging\n\nWhile we're on the subject of diagnostics, let's put some more in. A real\nchallenge I've found with garbage collectors is that they are opaque. We've been\nrunning lots of Lox programs just fine without any GC *at all* so far. Once we\nadd one, how do we tell if it's doing anything useful? Can we tell only if we\nwrite programs that plow through acres of memory? How do we debug that?\n\nAn easy way to shine a light into the GC's inner workings is with some logging.\n\n^code define-log-gc (1 before, 2 after)\n\nWhen this is enabled, clox prints information to the console when it does\nsomething with dynamic memory.\n\nWe need a couple of includes.\n\n^code debug-log-includes (1 before, 2 after)\n\nWe don't have a collector yet, but we can start putting in some of the logging\nnow. We'll want to know when a collection run starts.\n\n^code log-before-collect (1 before, 1 after)\n\nEventually we will log some other operations during the collection, so we'll\nalso want to know when the show's over.\n\n^code log-after-collect (2 before, 1 after)\n\nWe don't have any code for the collector yet, but we do have functions for\nallocating and freeing, so we can instrument those now.\n\n^code debug-log-allocate (1 before, 1 after)\n\nAnd at the end of an object's lifespan:\n\n^code log-free-object (1 before, 1 after)\n\nWith these two flags, we should be able to see that we're making progress as we\nwork through the rest of the chapter.\n\n## Marking the Roots\n\nObjects are scattered across the heap like stars in the inky night sky. A\nreference from one object to another forms a connection, and these\nconstellations are the graph that the mark phase traverses. Marking begins at\nthe roots.\n\n^code call-mark-roots (3 before, 2 after)\n\nMost roots are local variables or temporaries sitting right in the VM's stack,\nso we start by walking that.\n\n^code mark-roots\n\nTo mark a Lox value, we use this new function:\n\n^code mark-value-h (1 before, 1 after)\n\nIts implementation is here:\n\n^code mark-value\n\nSome Lox values -- numbers, Booleans, and `nil` -- are stored directly inline in\nValue and require no heap allocation. The garbage collector doesn't need to\nworry about them at all, so the first thing we do is ensure that the value is an\nactual heap object. If so, the real work happens in this function:\n\n^code mark-object-h (1 before, 1 after)\n\nWhich is defined here:\n\n^code mark-object\n\nThe `NULL` check is unnecessary when called from `markValue()`. A Lox Value that\nis some kind of Obj type will always have a valid pointer. But later we will\ncall this function directly from other code, and in some of those places, the\nobject being pointed to is optional.\n\nAssuming we do have a valid object, we mark it by setting a flag. That new field\nlives in the Obj header struct all objects share.\n\n^code is-marked-field (1 before, 1 after)\n\nEvery new object begins life unmarked because we haven't yet determined if it is\nreachable or not.\n\n^code init-is-marked (1 before, 2 after)\n\nBefore we go any farther, let's add some logging to `markObject()`.\n\n^code log-mark-object (2 before, 1 after)\n\nThis way we can see what the mark phase is doing. Marking the stack takes care\nof local variables and temporaries. The other main source of roots are the\nglobal variables.\n\n^code mark-globals (2 before, 1 after)\n\nThose live in a hash table owned by the VM, so we'll declare another helper\nfunction for marking all of the objects in a table.\n\n^code mark-table-h (2 before, 2 after)\n\nWe implement that in the \"table\" module here:\n\n^code mark-table\n\nPretty straightforward. We walk the entry array. For each one, we mark its\nvalue. We also mark the key strings for each entry since the GC manages those\nstrings too.\n\n### Less obvious roots\n\nThose cover the roots that we typically think of -- the values that are\nobviously reachable because they're stored in variables the user's program can\nsee. But the VM has a few of its own hidey-holes where it squirrels away\nreferences to values that it directly accesses.\n\nMost function call state lives in the value stack, but the VM maintains a\nseparate stack of CallFrames. Each CallFrame contains a pointer to the closure\nbeing called. The VM uses those pointers to access constants and upvalues, so\nthose closures need to be kept around too.\n\n^code mark-closures (1 before, 2 after)\n\nSpeaking of upvalues, the open upvalue list is another set of values that the\nVM can directly reach.\n\n^code mark-open-upvalues (3 before, 2 after)\n\nRemember also that a collection can begin during *any* allocation. Those\nallocations don't just happen while the user's program is running. The compiler\nitself periodically grabs memory from the heap for literals and the constant\ntable. If the GC runs while we're in the middle of compiling, then any values\nthe compiler directly accesses need to be treated as roots too.\n\nTo keep the compiler module cleanly separated from the rest of the VM, we'll do\nthat in a separate function.\n\n^code call-mark-compiler-roots (1 before, 1 after)\n\nIt's declared here:\n\n^code mark-compiler-roots-h (1 before, 2 after)\n\nWhich means the \"memory\" module needs an include.\n\n^code memory-include-compiler (2 before, 1 after)\n\nAnd the definition is over in the \"compiler\" module.\n\n^code mark-compiler-roots\n\nFortunately, the compiler doesn't have too many values that it hangs on to. The\nonly object it uses is the ObjFunction it is compiling into. Since function\ndeclarations can nest, the compiler has a linked list of those and we walk the\nwhole list.\n\nSince the \"compiler\" module is calling `markObject()`, it also needs an include.\n\n^code compiler-include-memory (1 before, 1 after)\n\nThose are all the roots. After running this, every object that the VM -- runtime\nand compiler -- can get to *without* going through some other object has its\nmark bit set.\n\n## Tracing Object References\n\nThe next step in the marking process is tracing through the graph of references\nbetween objects to find the indirectly reachable values. We don't have instances\nwith fields yet, so there aren't many objects that contain references, but we do\nhave some. In particular, ObjClosure has the list of\nObjUpvalues it closes over as well as a reference to the raw ObjFunction that it\nwraps. ObjFunction, in turn, has a constant table containing references to all\nof the literals created in the function's body. This is enough to build a fairly\ncomplex web of objects for our collector to crawl through.\n\n\n\nNow it's time to implement that traversal. We can go breadth-first, depth-first,\nor in some other order. Since we just need to find the *set* of all reachable\nobjects, the order we visit them mostly doesn't matter.\n\n\n\n### The tricolor abstraction\n\nAs the collector wanders through the graph of objects, we need to make sure it\ndoesn't lose track of where it is or get stuck going in circles. This is\nparticularly a concern for advanced implementations like incremental GCs that\ninterleave marking with running pieces of the user's program. The collector\nneeds to be able to pause and then pick up where it left off later.\n\nTo help us soft-brained humans reason about this complex process, VM hackers\ncame up with a metaphor called the **tricolor\nabstraction**. Each object has a conceptual \"color\" that tracks what state the\nobject is in, and what work is left to do.\n\n\n\n* **\"A White:** At the beginning of a garbage collection, every\n object is white. This color means we have not reached or processed the\n object at all.\n\n* **\"A Gray:** During marking, when we first reach an object, we\n darken it gray. This color means we know the object itself is reachable and\n should not be collected. But we have not yet traced *through* it to see what\n *other* objects it references. In graph algorithm terms, this is the\n *worklist* -- the set of objects we know about but haven't processed yet.\n\n* **\"A Black:** When\n we take a gray object and mark all of the objects it references, we then\n turn the gray object black. This color means the mark phase is done\n processing that object.\n\nIn terms of that abstraction, the marking process now looks like this:\n\n1. Start off with all objects white.\n\n2. Find all the roots and mark them gray.\n\n3. Repeat as long as there are still gray objects:\n\n 1. Pick a gray object. Turn any white objects that the object mentions to\n gray.\n\n 2. Mark the original gray object black.\n\nI find it helps to visualize this. You have a web of objects with references\nbetween them. Initially, they are all little white dots. Off to the side are\nsome incoming edges from the VM that point to the roots. Those roots turn gray.\nThen each gray object's siblings turn gray while the object itself turns black.\nThe full effect is a gray wavefront that passes through the graph, leaving a\nfield of reachable black objects behind it. Unreachable objects are not touched\nby the wavefront and stay white.\n\n\"A\n\nAt the end, you're left with a sea of reached,\nblack objects sprinkled with islands of white objects that can be swept up and\nfreed. Once the unreachable objects are freed, the remaining objects -- all\nblack -- are reset to white for the next garbage collection cycle.\n\n\n\n### A worklist for gray objects\n\nIn our implementation we have already marked the roots. They're all gray. The\nnext step is to start picking them and traversing their references. But we don't\nhave any easy way to find them. We set a field on the object, but that's it. We\ndon't want to have to traverse the entire object list looking for objects with\nthat field set.\n\nInstead, we'll create a separate worklist to keep track of all of the gray\nobjects. When an object turns gray, in addition to setting the mark field we'll\nalso add it to the worklist.\n\n^code add-to-gray-stack (1 before, 1 after)\n\nWe could use any kind of data structure that lets us put items in and take them\nout easily. I picked a stack because that's the simplest to implement with a\ndynamic array in C. It works mostly like other dynamic arrays we've built in\nLox, *except*, note that it calls the *system* `realloc()` function and not our\nown `reallocate()` wrapper. The memory for the gray stack itself is *not*\nmanaged by the garbage collector. We don't want growing the gray stack during a\nGC to cause the GC to recursively start a new GC. That could tear a hole in the\nspace-time continuum.\n\nWe'll manage its memory ourselves, explicitly. The VM owns the gray stack.\n\n^code vm-gray-stack (1 before, 1 after)\n\nIt starts out empty.\n\n^code init-gray-stack (1 before, 2 after)\n\nAnd we need to free it when the VM shuts down.\n\n^code free-gray-stack (2 before, 1 after)\n\nWe take full responsibility for this array. That\nincludes allocation failure. If we can't create or grow the gray stack, then we\ncan't finish the garbage collection. This is bad news for the VM, but\nfortunately rare since the gray stack tends to be pretty small. It would be nice\nto do something more graceful, but to keep the code in this book simple, we just\nabort.\n\n\n\n^code exit-gray-stack (2 before, 1 after)\n\n### Processing gray objects\n\nOK, now when we're done marking the roots, we have both set a bunch of fields\nand filled our work list with objects to chew through. It's time for the next\nphase.\n\n^code call-trace-references (1 before, 2 after)\n\nHere's the implementation:\n\n^code trace-references\n\nIt's as close to that textual algorithm as you can get. Until the stack empties,\nwe keep pulling out gray objects, traversing their references, and then marking\nthem black. Traversing an object's references may turn up new white objects that\nget marked gray and added to the stack. So this function swings back and forth\nbetween turning white objects gray and gray objects black, gradually advancing\nthe entire wavefront forward.\n\nHere's where we traverse a single object's references:\n\n^code blacken-object\n\nEach object kind has different fields that might\nreference other objects, so we need a specific blob of code for each type. We\nstart with the easy ones -- strings and native function objects contain no\noutgoing references so there is nothing to traverse.\n\n\n\nNote that we don't set any state in the traversed object itself. There is no\ndirect encoding of \"black\" in the object's state. A black object is any object\nwhose `isMarked` field is set and that is no longer in\nthe gray stack.\n\n\n\nNow let's start adding in the other object types. The simplest is upvalues.\n\n^code blacken-upvalue (2 before, 1 after)\n\nWhen an upvalue is closed, it contains a reference to the closed-over value.\nSince the value is no longer on the stack, we need to make sure we trace the\nreference to it from the upvalue.\n\nNext are functions.\n\n^code blacken-function (1 before, 1 after)\n\nEach function has a reference to an ObjString containing the function's name.\nMore importantly, the function has a constant table packed full of references to\nother objects. We trace all of those using this helper:\n\n^code mark-array\n\nThe last object type we have now -- we'll add more in later chapters -- is\nclosures.\n\n^code blacken-closure (1 before, 1 after)\n\nEach closure has a reference to the bare function it wraps, as well as an array\nof pointers to the upvalues it captures. We trace all of those.\n\nThat's the basic mechanism for processing a gray object, but there are two loose\nends to tie up. First, some logging.\n\n^code log-blacken-object (1 before, 1 after)\n\nThis way, we can watch the tracing percolate through the object graph. Speaking\nof which, note that I said *graph*. References between objects are directed, but\nthat doesn't mean they're *acyclic!* It's entirely possible to have cycles of\nobjects. When that happens, we need to ensure our collector doesn't get stuck in\nan infinite loop as it continually re-adds the same series of objects to the\ngray stack.\n\nThe fix is easy.\n\n^code check-is-marked (1 before, 1 after)\n\nIf the object is already marked, we don't mark it again and thus don't add it to\nthe gray stack. This ensures that an already-gray object is not redundantly\nadded and that a black object is not inadvertently turned back to gray. In other\nwords, it keeps the wavefront moving forward through only the white objects.\n\n## Sweeping Unused Objects\n\nWhen the loop in `traceReferences()` exits, we have processed all the objects we\ncould get our hands on. The gray stack is empty, and every object in the heap is\neither black or white. The black objects are reachable, and we want to hang on to\nthem. Anything still white never got touched by the trace and is thus garbage.\nAll that's left is to reclaim them.\n\n^code call-sweep (1 before, 2 after)\n\nAll of the logic lives in one function.\n\n^code sweep\n\nI know that's kind of a lot of code and pointer shenanigans, but there isn't\nmuch to it once you work through it. The outer `while` loop walks the linked\nlist of every object in the heap, checking their mark bits. If an object is\nmarked (black), we leave it alone and continue past it. If it is unmarked\n(white), we unlink it from the list and free it using the `freeObject()`\nfunction we already wrote.\n\n\"A\n\nMost of the other code in here deals with the fact that removing a node from a\nsingly linked list is cumbersome. We have to continuously remember the previous\nnode so we can unlink its next pointer, and we have to handle the edge case\nwhere we are freeing the first node. But, otherwise, it's pretty simple --\ndelete every node in a linked list that doesn't have a bit set in it.\n\nThere's one little addition:\n\n^code unmark (1 before, 1 after)\n\nAfter `sweep()` completes, the only remaining objects are the live black ones\nwith their mark bits set. That's correct, but when the *next* collection cycle\nstarts, we need every object to be white. So whenever we reach a black object,\nwe go ahead and clear the bit now in anticipation of the next run.\n\n### Weak references and the string pool\n\nWe are almost done collecting. There is one remaining corner of the VM that has\nsome unusual requirements around memory. Recall that when we added strings to\nclox we made the VM intern them all. That means the VM has a hash table\ncontaining a pointer to every single string in the heap. The VM uses this to\nde-duplicate strings.\n\nDuring the mark phase, we deliberately did *not* treat the VM's string table as\na source of roots. If we had, no string would *ever*\nbe collected. The string table would grow and grow and never yield a single byte\nof memory back to the operating system. That would be bad.\n\n\n\nAt the same time, if we *do* let the GC free strings, then the VM's string table\nwill be left with dangling pointers to freed memory. That would be even worse.\n\nThe string table is special and we need special support for it. In particular,\nit needs a special kind of reference. The table should be able to refer to a\nstring, but that link should not be considered a root when determining\nreachability. That implies that the referenced object can be freed. When that\nhappens, the dangling reference must be fixed too, sort of like a magic,\nself-clearing pointer. This particular set of semantics comes up frequently\nenough that it has a name: a [**weak reference**][weak].\n\n[weak]: https://en.wikipedia.org/wiki/Weak_reference\n\nWe have already implicitly implemented half of the string table's unique\nbehavior by virtue of the fact that we *don't* traverse it during marking. That\nmeans it doesn't force strings to be reachable. The remaining piece is clearing\nout any dangling pointers for strings that are freed.\n\nTo remove references to unreachable strings, we need to know which strings *are*\nunreachable. We don't know that until after the mark phase has completed. But we\ncan't wait until after the sweep phase is done because by then the objects --\nand their mark bits -- are no longer around to check. So the right time is\nexactly between the marking and sweeping phases.\n\n^code sweep-strings (1 before, 1 after)\n\nThe logic for removing the about-to-be-deleted strings exists in a new function\nin the \"table\" module.\n\n^code table-remove-white-h (2 before, 2 after)\n\nThe implementation is here:\n\n^code table-remove-white\n\nWe walk every entry in the table. The string intern table uses only the key of\neach entry -- it's basically a hash *set* not a hash *map*. If the key string\nobject's mark bit is not set, then it is a white object that is moments from\nbeing swept away. We delete it from the hash table first and thus ensure we\nwon't see any dangling pointers.\n\n## When to Collect\n\nWe have a fully functioning mark-sweep garbage collector now. When the stress\ntesting flag is enabled, it gets called all the time, and with the logging\nenabled too, we can watch it do its thing and see that it is indeed reclaiming\nmemory. But, when the stress testing flag is off, it never runs at all. It's\ntime to decide when the collector should be invoked during normal program\nexecution.\n\nAs far as I can tell, this question is poorly answered by the literature. When\ngarbage collectors were first invented, computers had a tiny, fixed amount of\nmemory. Many of the early GC papers assumed that you set aside a few thousand\nwords of memory -- in other words, most of it -- and invoked the collector\nwhenever you ran out. Simple.\n\nModern machines have gigs of physical RAM, hidden behind the operating system's\neven larger virtual memory abstraction, which is shared among a slew of other\nprograms all fighting for their chunk of memory. The operating system will let\nyour program request as much as it wants and then page in and out from the disc\nwhen physical memory gets full. You never really \"run out\" of memory, you just\nget slower and slower.\n\n### Latency and throughput\n\nIt no longer makes sense to wait until you \"have to\", to run the GC, so we need\na more subtle timing strategy. To reason about this more precisely, it's time to\nintroduce two fundamental numbers used when measuring a memory manager's\nperformance: *throughput* and *latency*.\n\nEvery managed language pays a performance price compared to explicit,\nuser-authored deallocation. The time spent actually freeing memory is the same,\nbut the GC spends cycles figuring out *which* memory to free. That is time *not*\nspent running the user's code and doing useful work. In our implementation,\nthat's the entirety of the mark phase. The goal of a sophisticated garbage\ncollector is to minimize that overhead.\n\nThere are two key metrics we can use to understand that cost better:\n\n* **Throughput** is the total fraction of time spent running user code versus\n doing garbage collection work. Say you run a clox program for ten seconds\n and it spends a second of that inside `collectGarbage()`. That means the\n throughput is 90% -- it spent 90% of the time running the program and 10%\n on GC overhead.\n\n Throughput is the most fundamental measure because it tracks the total cost\n of collection overhead. All else being equal, you want to maximize\n throughput. Up until this chapter, clox had no GC at all and thus 100% throughput. That's pretty hard to beat. Of\n course, it came at the slight expense of potentially running out of memory\n and crashing if the user's program ran long enough. You can look at the goal\n of a GC as fixing that \"glitch\" while sacrificing as little throughput as\n possible.\n\n\n\n* **Latency** is the longest *continuous* chunk of time where the user's\n program is completely paused while garbage collection happens. It's a\n measure of how \"chunky\" the collector is. Latency is an entirely different\n metric than throughput.\n\n Consider two runs of a clox program that both take ten seconds. In the first\n run, the GC kicks in once and spends a solid second in `collectGarbage()` in\n one massive collection. In the second run, the GC gets invoked five times,\n each for a fifth of a second. The *total* amount of time spent collecting is\n still a second, so the throughput is 90% in both cases. But in the second\n run, the latency is only 1/5th of a second, five times less than in the\n first.\n\n\n\n\"A\n\n\n\nIf you like analogies, imagine your program is a bakery selling fresh-baked\nbread to customers. Throughput is the total number of warm, crusty baguettes you\ncan serve to customers in a single day. Latency is how long the unluckiest\ncustomer has to wait in line before they get served.\n\nRunning the garbage collector is like shutting\ndown the bakery temporarily to go through all of the dishes, sort out the dirty\nfrom the clean, and then wash the used ones. In our analogy, we don't have\ndedicated dishwashers, so while this is going on, no baking is happening. The\nbaker is washing up.\n\n\n\nSelling fewer loaves of bread a day is bad, and making any particular customer\nsit and wait while you clean all the dishes is too. The goal is to maximize\nthroughput and minimize latency, but there is no free lunch, even inside a\nbakery. Garbage collectors make different trade-offs between how much throughput\nthey sacrifice and latency they tolerate.\n\nBeing able to make these trade-offs is useful because different user programs\nhave different needs. An overnight batch job that is generating a report from a\nterabyte of data just needs to get as much work done as fast as possible.\nThroughput is queen. Meanwhile, an app running on a user's smartphone needs to\nalways respond immediately to user input so that dragging on the screen feels\nbuttery smooth. The app can't freeze for a few\nseconds while the GC mucks around in the heap.\n\n\n\nAs a garbage collector author, you control some of the trade-off between\nthroughput and latency by your choice of collection algorithm. But even within a\nsingle algorithm, we have a lot of control over *how frequently* the collector\nruns.\n\nOur collector is a **stop-the-world GC** which\nmeans the user's program is paused until the entire garbage collection process\nhas completed. If we wait a long time before we run the collector, then a large\nnumber of dead objects will accumulate. That leads to a very long pause while\nthe collector runs, and thus high latency. So, clearly, we want to run the\ncollector really frequently.\n\n\n\nBut every time the collector runs, it spends some time visiting live objects.\nThat doesn't really *do* anything useful (aside from ensuring that they don't\nincorrectly get deleted). Time visiting live objects is time not freeing memory\nand also time not running user code. If you run the GC *really* frequently, then\nthe user's program doesn't have enough time to even generate new garbage for the\nVM to collect. The VM will spend all of its time obsessively revisiting the same\nset of live objects over and over, and throughput will suffer. So, clearly, we\nwant to run the collector really *in*frequently.\n\nIn fact, we want something in the middle, and the frequency of when the\ncollector runs is one of our main knobs for tuning the trade-off between latency\nand throughput.\n\n### Self-adjusting heap\n\nWe want our GC to run frequently enough to minimize latency but infrequently\nenough to maintain decent throughput. But how do we find the balance between\nthese when we have no idea how much memory the user's program needs and how\noften it allocates? We could pawn the problem onto the user and force them to\npick by exposing GC tuning parameters. Many VMs do this. But if we, the GC\nauthors, don't know how to tune it well, odds are good most users won't either.\nThey deserve a reasonable default behavior.\n\nI'll be honest with you, this is not my area of expertise. I've talked to a\nnumber of professional GC hackers -- this is something you can build an entire\ncareer on -- and read a lot of the literature, and all of the answers I got\nwere... vague. The strategy I ended up picking is common, pretty simple, and (I\nhope!) good enough for most uses.\n\nThe idea is that the collector frequency automatically adjusts based on the live\nsize of the heap. We track the total number of bytes of managed memory that the\nVM has allocated. When it goes above some threshold, we trigger a GC. After\nthat, we note how many bytes of memory remain -- how many were *not* freed. Then\nwe adjust the threshold to some value larger than that.\n\nThe result is that as the amount of live memory increases, we collect less\nfrequently in order to avoid sacrificing throughput by re-traversing the growing\npile of live objects. As the amount of live memory goes down, we collect more\nfrequently so that we don't lose too much latency by waiting too long.\n\nThe implementation requires two new bookkeeping fields in the VM.\n\n^code vm-fields (1 before, 1 after)\n\nThe first is a running total of the number of bytes of managed memory the VM has\nallocated. The second is the threshold that triggers the next collection. We\ninitialize them when the VM starts up.\n\n^code init-gc-fields (1 before, 2 after)\n\nThe starting threshold here is arbitrary. It's similar\nto the initial capacity we picked for our various dynamic arrays. The goal is to\nnot trigger the first few GCs *too* quickly but also to not wait too long. If we\nhad some real-world Lox programs, we could profile those to tune this. But since\nall we have are toy programs, I just picked a number.\n\n\n\nEvery time we allocate or free some memory, we adjust the counter by that delta.\n\n^code updated-bytes-allocated (1 before, 1 after)\n\nWhen the total crosses the limit, we run the collector.\n\n^code collect-on-next (2 before, 1 after)\n\nNow, finally, our garbage collector actually does something when the user runs a\nprogram without our hidden diagnostic flag enabled. The sweep phase frees\nobjects by calling `reallocate()`, which lowers the value of `bytesAllocated`,\nso after the collection completes, we know how many live bytes remain. We adjust\nthe threshold of the next GC based on that.\n\n^code update-next-gc (1 before, 2 after)\n\nThe threshold is a multiple of the heap size. This way, as the amount of memory\nthe program uses grows, the threshold moves farther out to limit the total time\nspent re-traversing the larger live set. Like other numbers in this chapter, the\nscaling factor is basically arbitrary.\n\n^code heap-grow-factor (1 before, 2 after)\n\nYou'd want to tune this in your implementation once you had some real programs\nto benchmark it on. Right now, we can at least log some of the statistics that\nwe have. We capture the heap size before the collection.\n\n^code log-before-size (1 before, 1 after)\n\nAnd then print the results at the end.\n\n^code log-collected-amount (1 before, 1 after)\n\nThis way we can see how much the garbage collector accomplished while it ran.\n\n## Garbage Collection Bugs\n\nIn theory, we are all done now. We have a GC. It kicks in periodically, collects\nwhat it can, and leaves the rest. If this were a typical textbook, we would wipe\nthe dust from our hands and bask in the soft glow of the flawless marble edifice\nwe have created.\n\nBut I aim to teach you not just the theory of programming languages but the\nsometimes painful reality. I am going to roll over a rotten log and show you the\nnasty bugs that live under it, and garbage collector bugs really are some of the\ngrossest invertebrates out there.\n\nThe collector's job is to free dead objects and preserve live ones. Mistakes are\neasy to make in both directions. If the VM fails to free objects that aren't\nneeded, it slowly leaks memory. If it frees an object that is in use, the user's\nprogram can access invalid memory. These failures often don't immediately cause\na crash, which makes it hard for us to trace backward in time to find the bug.\n\nThis is made harder by the fact that we don't know when the collector will run.\nAny call that eventually allocates some memory is a place in the VM where a\ncollection could happen. It's like musical chairs. At any point, the GC might\nstop the music. Every single heap-allocated object that we want to keep needs to\nfind a chair quickly -- get marked as a root or stored as a reference in some\nother object -- before the sweep phase comes to kick it out of the game.\n\nHow is it possible for the VM to use an object later -- one that the GC itself\ndoesn't see? How can the VM find it? The most common answer is through a pointer\nstored in some local variable on the C stack. The GC walks the *VM's* value and\nCallFrame stacks, but the C stack is hidden to it.\n\n\n\nIn previous chapters, we wrote seemingly pointless code that pushed an object\nonto the VM's value stack, did a little work, and then popped it right back off.\nMost times, I said this was for the GC's benefit. Now you see why. The code\nbetween pushing and popping potentially allocates memory and thus can trigger a\nGC. We had to make sure the object was on the value stack so that the\ncollector's mark phase would find it and keep it alive.\n\nI wrote the entire clox implementation before splitting it into chapters and\nwriting the prose, so I had plenty of time to find all of these corners and\nflush out most of these bugs. The stress testing code we put in at the beginning\nof this chapter and a pretty good test suite were very helpful.\n\nBut I fixed only *most* of them. I left a couple in because I want to give you a\nhint of what it's like to encounter these bugs in the wild. If you enable the\nstress test flag and run some toy Lox programs, you can probably stumble onto a\nfew. Give it a try and *see if you can fix any yourself*.\n\n\n### Adding to the constant table\n\nYou are very likely to hit the first bug. The constant table each chunk owns is\na dynamic array. When the compiler adds a new constant to the current function's\ntable, that array may need to grow. The constant itself may also be some\nheap-allocated object like a string or a nested function.\n\nThe new object being added to the constant table is passed to `addConstant()`.\nAt that moment, the object can be found only in the parameter to that function\non the C stack. That function appends the object to the constant table. If the\ntable doesn't have enough capacity and needs to grow, it calls `reallocate()`.\nThat in turn triggers a GC, which fails to mark the new constant object and\nthus sweeps it right before we have a chance to add it to the table. Crash.\n\nThe fix, as you've seen in other places, is to push the constant onto the stack\ntemporarily.\n\n^code add-constant-push (1 before, 1 after)\n\nOnce the constant table contains the object, we pop it off the stack.\n\n^code add-constant-pop (1 before, 1 after)\n\nWhen the GC is marking roots, it walks the chain of compilers and marks each of\ntheir functions, so the new constant is reachable now. We do need an include\nto call into the VM from the \"chunk\" module.\n\n^code chunk-include-vm (1 before, 2 after)\n\n### Interning strings\n\nHere's another similar one. All strings are interned in clox, so whenever we\ncreate a new string, we also add it to the intern table. You can see where this\nis going. Since the string is brand new, it isn't reachable anywhere. And\nresizing the string pool can trigger a collection. Again, we go ahead and stash\nthe string on the stack first.\n\n^code push-string (2 before, 1 after)\n\nAnd then pop it back off once it's safely nestled in the table.\n\n^code pop-string (1 before, 2 after)\n\nThis ensures the string is safe while the table is being resized. Once it\nsurvives that, `allocateString()` will return it to some caller which can then\ntake responsibility for ensuring the string is still reachable before the next\nheap allocation occurs.\n\n### Concatenating strings\n\nOne last example: Over in the interpreter, the `OP_ADD` instruction can be used\nto concatenate two strings. As it does with numbers, it pops the two operands\nfrom the stack, computes the result, and pushes that new value back onto the\nstack. For numbers that's perfectly safe.\n\nBut concatenating two strings requires allocating a new character array on the\nheap, which can in turn trigger a GC. Since we've already popped the operand\nstrings by that point, they can potentially be missed by the mark phase and get\nswept away. Instead of popping them off the stack eagerly, we peek them.\n\n^code concatenate-peek (1 before, 2 after)\n\nThat way, they are still hanging out on the stack when we create the result\nstring. Once that's done, we can safely pop them off and replace them with the\nresult.\n\n^code concatenate-pop (1 before, 1 after)\n\nThose were all pretty easy, especially because I *showed* you where the fix was.\nIn practice, *finding* them is the hard part. All you see is an object that\n*should* be there but isn't. It's not like other bugs where you're looking for\nthe code that *causes* some problem. You're looking for the *absence* of code\nwhich fails to *prevent* a problem, and that's a much harder search.\n\nBut, for now at least, you can rest easy. As far as I know, we've found all of\nthe collection bugs in clox, and now we have a working, robust, self-tuning,\nmark-sweep garbage collector.\n\n
\n\n## Challenges\n\n1. The Obj header struct at the top of each object now has three fields:\n `type`, `isMarked`, and `next`. How much memory do those take up (on your\n machine)? Can you come up with something more compact? Is there a runtime\n cost to doing so?\n\n1. When the sweep phase traverses a live object, it clears the `isMarked`\n field to prepare it for the next collection cycle. Can you come up with a\n more efficient approach?\n\n1. Mark-sweep is only one of a variety of garbage collection algorithms out\n there. Explore those by replacing or augmenting the current collector with\n another one. Good candidates to consider are reference counting, Cheney's\n algorithm, or the Lisp 2 mark-compact algorithm.\n\n
\n\n
\n\n## Design Note: Generational Collectors\n\nA collector loses throughput if it spends a long time re-visiting objects that\nare still alive. But it can increase latency if it avoids collecting and\naccumulates a large pile of garbage to wade through. If only there were some way\nto tell which objects were likely to be long-lived and which weren't. Then the\nGC could avoid revisiting the long-lived ones as often and clean up the\nephemeral ones more frequently.\n\nIt turns out there kind of is. Many years ago, GC researchers gathered metrics\non the lifetime of objects in real-world running programs. They tracked every\nobject when it was allocated, and eventually when it was no longer needed, and\nthen graphed out how long objects tended to live.\n\nThey discovered something they called the **generational hypothesis**, or the\nmuch less tactful term **infant mortality**. Their observation was that most\nobjects are very short-lived but once they survive beyond a certain age, they\ntend to stick around quite a long time. The longer an object *has* lived, the\nlonger it likely will *continue* to live. This observation is powerful because\nit gave them a handle on how to partition objects into groups that benefit from\nfrequent collections and those that don't.\n\nThey designed a technique called **generational garbage collection**. It works\nlike this: Every time a new object is allocated, it goes into a special,\nrelatively small region of the heap called the \"nursery\". Since objects tend to\ndie young, the garbage collector is invoked frequently over the objects just in this region.\n\n\n\nEach time the GC runs over the nursery is called a \"generation\". Any objects\nthat are no longer needed get freed. Those that survive are now considered one\ngeneration older, and the GC tracks this for each object. If an object survives\na certain number of generations -- often just a single collection -- it gets\n*tenured*. At this point, it is copied out of the nursery into a much larger\nheap region for long-lived objects. The garbage collector runs over that region\ntoo, but much less frequently since odds are good that most of those objects\nwill still be alive.\n\nGenerational collectors are a beautiful marriage of empirical data -- the\nobservation that object lifetimes are *not* evenly distributed -- and clever\nalgorithm design that takes advantage of that fact. They're also conceptually\nquite simple. You can think of one as just two separately tuned GCs and a pretty\nsimple policy for moving objects from one to the other.\n\n
\n" }, { "path": "book/global-variables.md", "content": "> If only there could be an invention that bottled up a memory, like scent. And\n> it never faded, and it never got stale. And then, when one wanted it, the\n> bottle could be uncorked, and it would be like living the moment all over\n> again.\n>\n> Daphne du Maurier, Rebecca\n\nThe [previous chapter][hash] was a long exploration of one big, deep,\nfundamental computer science data structure. Heavy on theory and concept. There\nmay have been some discussion of big-O notation and algorithms. This chapter has\nfewer intellectual pretensions. There are no large ideas to learn. Instead, it's\na handful of straightforward engineering tasks. Once we've completed them, our\nvirtual machine will support variables.\n\nActually, it will support only *global* variables. Locals are coming in the\n[next chapter][]. In jlox, we managed to cram them both into a single chapter\nbecause we used the same implementation technique for all variables. We built a\nchain of environments, one for each scope, all the way up to the top. That was a\nsimple, clean way to learn how to manage state.\n\n[next chapter]: local-variables.html\n\nBut it's also *slow*. Allocating a new hash table each time you enter a block or\ncall a function is not the road to a fast VM. Given how much code is concerned\nwith using variables, if variables go slow, everything goes slow. For clox,\nwe'll improve that by using a much more efficient strategy for local variables, but globals aren't as easily optimized.\n\n\n\n[hash]: hash-tables.html\n\nA quick refresher on Lox semantics: Global variables in Lox are \"late bound\", or\nresolved dynamically. This means you can compile a chunk of code that refers to\na global variable before it's defined. As long as the code doesn't *execute*\nbefore the definition happens, everything is fine. In practice, that means you\ncan refer to later variables inside the body of functions.\n\n```lox\nfun showVariable() {\n print global;\n}\n\nvar global = \"after\";\nshowVariable();\n```\n\nCode like this might seem odd, but it's handy for defining mutually recursive\nfunctions. It also plays nicer with the REPL. You can write a little function in\none line, then define the variable it uses in the next.\n\nLocal variables work differently. Since a local variable's declaration *always*\noccurs before it is used, the VM can resolve them at compile time, even in a\nsimple single-pass compiler. That will let us use a smarter representation for\nlocals. But that's for the next chapter. Right now, let's just worry about\nglobals.\n\n## Statements\n\nVariables come into being using variable declarations, which means now is also\nthe time to add support for statements to our compiler. If you recall, Lox\nsplits statements into two categories. \"Declarations\" are those statements that\nbind a new name to a value. The other kinds of statements -- control flow,\nprint, etc. -- are just called \"statements\". We disallow declarations directly\ninside control flow statements, like this:\n\n```lox\nif (monday) var croissant = \"yes\"; // Error.\n```\n\nAllowing it would raise confusing questions around the scope of the variable.\nSo, like other languages, we prohibit it syntactically by having a separate\ngrammar rule for the subset of statements that *are* allowed inside a control\nflow body.\n\n```ebnf\nstatement → exprStmt\n | forStmt\n | ifStmt\n | printStmt\n | returnStmt\n | whileStmt\n | block ;\n```\n\nThen we use a separate rule for the top level of a script and inside a block.\n\n```ebnf\ndeclaration → classDecl\n | funDecl\n | varDecl\n | statement ;\n```\n\nThe `declaration` rule contains the statements that declare names, and also\nincludes `statement` so that all statement types are allowed. Since `block`\nitself is in `statement`, you can put declarations inside a control flow construct by nesting them inside a\nblock.\n\n\n\nIn this chapter, we'll cover only a couple of statements and one\ndeclaration.\n\n```ebnf\nstatement → exprStmt\n | printStmt ;\n\ndeclaration → varDecl\n | statement ;\n```\n\nUp to now, our VM considered a \"program\" to be a single expression since that's\nall we could parse and compile. In a full Lox implementation, a program is a\nsequence of declarations. We're ready to support that now.\n\n^code compile (1 before, 1 after)\n\nWe keep compiling declarations until we hit the end of the source file. We\ncompile a single declaration using this:\n\n^code declaration\n\nWe'll get to variable declarations later in the chapter, so for now, we simply\nforward to `statement()`.\n\n^code statement\n\nBlocks can contain declarations, and control flow statements can contain other\nstatements. That means these two functions will eventually be recursive. We may\nas well write out the forward declarations now.\n\n^code forward-declarations (1 before, 1 after)\n\n### Print statements\n\nWe have two statement types to support in this chapter. Let's start with `print`\nstatements, which begin, naturally enough, with a `print` token. We detect that\nusing this helper function:\n\n^code match\n\nYou may recognize it from jlox. If the current token has the given type, we\nconsume the token and return `true`. Otherwise we leave the token alone and\nreturn `false`. This helper function is implemented\nin terms of this other helper:\n\n\n\n^code check\n\nThe `check()` function returns `true` if the current token has the given type.\nIt seems a little silly to wrap this in a function, but\nwe'll use it more later, and I think short verb-named functions like this make\nthe parser easier to read.\n\n\n\nIf we did match the `print` token, then we compile the rest of the statement\nhere:\n\n^code print-statement\n\nA `print` statement evaluates an expression and prints the result, so we first\nparse and compile that expression. The grammar expects a semicolon after that,\nso we consume it. Finally, we emit a new instruction to print the result.\n\n^code op-print (1 before, 1 after)\n\nAt runtime, we execute this instruction like so:\n\n^code interpret-print (1 before, 1 after)\n\nWhen the interpreter reaches this instruction, it has already executed the code\nfor the expression, leaving the result value on top of the stack. Now we simply\npop and print it.\n\nNote that we don't push anything else after that. This is a key difference\nbetween expressions and statements in the VM. Every bytecode instruction has a\n**stack effect** that describes how the instruction\nmodifies the stack. For example, `OP_ADD` pops two values and pushes one,\nleaving the stack one element smaller than before.\n\n\n\nYou can sum the stack effects of a series of instructions to get their total\neffect. When you add the stack effects of the series of instructions compiled\nfrom any complete expression, it will total one. Each expression leaves one\nresult value on the stack.\n\nThe bytecode for an entire statement has a total stack effect of zero. Since a\nstatement produces no values, it ultimately leaves the stack unchanged, though\nit of course uses the stack while it's doing its thing. This is important\nbecause when we get to control flow and looping, a program might execute a long\nseries of statements. If each statement grew or shrank the stack, it might\neventually overflow or underflow.\n\nWhile we're in the interpreter loop, we should delete a bit of code.\n\n^code op-return (1 before, 1 after)\n\nWhen the VM only compiled and evaluated a single expression, we had some\ntemporary code in `OP_RETURN` to output the value. Now that we have statements\nand `print`, we don't need that anymore. We're one step closer to the complete implementation of clox.\n\n\n\nAs usual, a new instruction needs support in the disassembler.\n\n^code disassemble-print (1 before, 1 after)\n\nThat's our `print` statement. If you want, give it a whirl:\n\n```lox\nprint 1 + 2;\nprint 3 * 4;\n```\n\nExciting! OK, maybe not thrilling, but we can build scripts that contain as many\nstatements as we want now, which feels like progress.\n\n### Expression statements\n\nWait until you see the next statement. If we *don't* see a `print` keyword, then\nwe must be looking at an expression statement.\n\n^code parse-expressions-statement (1 before, 1 after)\n\nIt's parsed like so:\n\n^code expression-statement\n\nAn \"expression statement\" is simply an expression followed by a semicolon.\nThey're how you write an expression in a context where a statement is expected.\nUsually, it's so that you can call a function or evaluate an assignment for its\nside effect, like this:\n\n```lox\nbrunch = \"quiche\";\neat(brunch);\n```\n\nSemantically, an expression statement evaluates the expression and discards the\nresult. The compiler directly encodes that behavior. It compiles the expression,\nand then emits an `OP_POP` instruction.\n\n^code pop-op (1 before, 1 after)\n\nAs the name implies, that instruction pops the top value off the stack and\nforgets it.\n\n^code interpret-pop (1 before, 1 after)\n\nWe can disassemble it too.\n\n^code disassemble-pop (1 before, 1 after)\n\nExpression statements aren't very useful yet since we can't create any\nexpressions that have side effects, but they'll be essential when we\n[add functions later][functions]. The majority of\nstatements in real-world code in languages like C are expression statements.\n\n\n\n[functions]: calls-and-functions.html\n\n### Error synchronization\n\nWhile we're getting this initial work done in the compiler, we can tie off a\nloose end we left [several chapters back][errors]. Like jlox, clox uses panic\nmode error recovery to minimize the number of cascaded compile errors that it\nreports. The compiler exits panic mode when it reaches a synchronization point.\nFor Lox, we chose statement boundaries as that point. Now that we have\nstatements, we can implement synchronization.\n\n[errors]: compiling-expressions.html#handling-syntax-errors\n\n^code call-synchronize (1 before, 1 after)\n\nIf we hit a compile error while parsing the previous statement, we enter panic\nmode. When that happens, after the statement we start synchronizing.\n\n^code synchronize\n\nWe skip tokens indiscriminately until we reach something that looks like a\nstatement boundary. We recognize the boundary by looking for a preceding token\nthat can end a statement, like a semicolon. Or we'll look for a subsequent token\nthat begins a statement, usually one of the control flow or declaration\nkeywords.\n\n## Variable Declarations\n\nMerely being able to *print* doesn't win your language any prizes at the\nprogramming language fair, so let's move on to\nsomething a little more ambitious and get variables going. There are three\noperations we need to support:\n\n\n\n* Declaring a new variable using a `var` statement.\n* Accessing the value of a variable using an identifier expression.\n* Storing a new value in an existing variable using an assignment expression.\n\nWe can't do either of the last two until we have some variables, so we start\nwith declarations.\n\n^code match-var (1 before, 2 after)\n\nThe placeholder parsing function we sketched out for the declaration grammar\nrule has an actual production now. If we match a `var` token, we jump here:\n\n^code var-declaration\n\nThe keyword is followed by the variable name. That's compiled by\n`parseVariable()`, which we'll get to in a second. Then we look for an `=`\nfollowed by an initializer expression. If the user doesn't initialize the\nvariable, the compiler implicitly initializes it to `nil` by emitting an `OP_NIL` instruction. Either way, we\nexpect the statement to be terminated with a semicolon.\n\n\n\nThere are two new functions here for working with variables and identifiers.\nHere is the first:\n\n^code parse-variable (2 before)\n\nIt requires the next token to be an identifier, which it consumes and sends\nhere:\n\n^code identifier-constant (2 before)\n\nThis function takes the given token and adds its lexeme to the chunk's constant\ntable as a string. It then returns the index of that constant in the constant\ntable.\n\nGlobal variables are looked up *by name* at runtime. That means the VM -- the\nbytecode interpreter loop -- needs access to the name. A whole string is too big\nto stuff into the bytecode stream as an operand. Instead, we store the string in\nthe constant table and the instruction then refers to the name by its index in\nthe table.\n\nThis function returns that index all the way to `varDeclaration()` which later\nhands it over to here:\n\n^code define-variable\n\nThis outputs the bytecode instruction that defines\nthe new variable and stores its initial value. The index of the variable's name\nin the constant table is the instruction's operand. As usual in a stack-based\nVM, we emit this instruction last. At runtime, we execute the code for the\nvariable's initializer first. That leaves the value on the stack. Then this\ninstruction takes that value and stores it away for later.\n\n\n\nOver in the runtime, we begin with this new instruction:\n\n^code define-global-op (1 before, 1 after)\n\nThanks to our handy-dandy hash table, the implementation isn't too hard.\n\n^code interpret-define-global (1 before, 1 after)\n\nWe get the name of the variable from the constant table. Then we take the value from the top of the stack and store it in a\nhash table with that name as the key.\n\n\n\nThis code doesn't check to see if the key is already in the table. Lox is pretty\nlax with global variables and lets you redefine them without error. That's\nuseful in a REPL session, so the VM supports that by simply overwriting the\nvalue if the key happens to already be in the hash table.\n\nThere's another little helper macro:\n\n^code read-string (1 before, 1 after)\n\nIt reads a one-byte operand from the bytecode chunk. It treats that as an index\ninto the chunk's constant table and returns the string at that index. It doesn't\ncheck that the value *is* a string -- it just indiscriminately casts it. That's\nsafe because the compiler never emits an instruction that refers to a non-string\nconstant.\n\nBecause we care about lexical hygiene, we also undefine this macro at the end of\nthe interpret function.\n\n^code undef-read-string (1 before, 1 after)\n\nI keep saying \"the hash table\", but we don't actually have one yet. We need a\nplace to store these globals. Since we want them to persist as long as clox is\nrunning, we store them right in the VM.\n\n^code vm-globals (1 before, 1 after)\n\nAs we did with the string table, we need to initialize the hash table to a valid\nstate when the VM boots up.\n\n^code init-globals (1 before, 1 after)\n\nAnd we tear it down when we exit.\n\n\n\n^code free-globals (1 before, 1 after)\n\nAs usual, we want to be able to disassemble the new instruction too.\n\n^code disassemble-define-global (1 before, 1 after)\n\nAnd with that, we can define global variables. Not that users can *tell* that\nthey've done so, because they can't actually *use* them. So let's fix that next.\n\n## Reading Variables\n\nAs in every programming language ever, we access a variable's value using its\nname. We hook up identifier tokens to the expression parser here:\n\n^code table-identifier (1 before, 1 after)\n\nThat calls this new parser function:\n\n^code variable-without-assign\n\nLike with declarations, there are a couple of tiny helper functions that seem\npointless now but will become more useful in later chapters. I promise.\n\n^code read-named-variable\n\nThis calls the same `identifierConstant()` function from before to take the\ngiven identifier token and add its lexeme to the chunk's constant table as a\nstring. All that remains is to emit an instruction that loads the global\nvariable with that name. Here's the instruction:\n\n^code get-global-op (1 before, 1 after)\n\nOver in the interpreter, the implementation mirrors `OP_DEFINE_GLOBAL`.\n\n^code interpret-get-global (1 before, 1 after)\n\nWe pull the constant table index from the instruction's operand and get the\nvariable name. Then we use that as a key to look up the variable's value in the\nglobals hash table.\n\nIf the key isn't present in the hash table, it means that global variable has\nnever been defined. That's a runtime error in Lox, so we report it and exit the\ninterpreter loop if that happens. Otherwise, we take the value and push it\nonto the stack.\n\n^code disassemble-get-global (1 before, 1 after)\n\nA little bit of disassembling, and we're done. Our interpreter is now able to\nrun code like this:\n\n```lox\nvar beverage = \"cafe au lait\";\nvar breakfast = \"beignets with \" + beverage;\nprint breakfast;\n```\n\nThere's only one operation left.\n\n## Assignment\n\nThroughout this book, I've tried to keep you on a fairly safe and easy path. I\ndon't avoid hard *problems*, but I try to not make the *solutions* more complex\nthan they need to be. Alas, other design choices in our bytecode compiler make assignment annoying to implement.\n\n\n\nOur bytecode VM uses a single-pass compiler. It parses and generates bytecode\non the fly without any intermediate AST. As soon as it recognizes a piece of\nsyntax, it emits code for it. Assignment doesn't naturally fit that. Consider:\n\n```lox\nmenu.brunch(sunday).beverage = \"mimosa\";\n```\n\nIn this code, the parser doesn't realize `menu.brunch(sunday).beverage` is the\ntarget of an assignment and not a normal expression until it reaches `=`, many\ntokens after the first `menu`. By then, the compiler has already emitted\nbytecode for the whole thing.\n\nThe problem is not as dire as it might seem, though. Look at how the parser sees that example:\n\n\"The\n\nEven though the `.beverage` part must not be compiled as a get expression,\neverything to the left of the `.` is an expression, with the normal expression\nsemantics. The `menu.brunch(sunday)` part can be compiled and executed as usual.\n\nFortunately for us, the only semantic differences on the left side of an\nassignment appear at the very right-most end of the tokens, immediately\npreceding the `=`. Even though the receiver of a setter may be an arbitrarily\nlong expression, the part whose behavior differs from a get expression is only\nthe trailing identifier, which is right before the `=`. We don't need much\nlookahead to realize `beverage` should be compiled as a set expression and not a\ngetter.\n\nVariables are even easier since they are just a single bare identifier before an\n`=`. The idea then is that right *before* compiling an expression that can also\nbe used as an assignment target, we look for a subsequent `=` token. If we see\none, we compile it as an assignment or setter instead of a variable access or\ngetter.\n\nWe don't have setters to worry about yet, so all we need to handle are variables.\n\n^code named-variable (1 before, 1 after)\n\nIn the parse function for identifier expressions, we look for an equals sign\nafter the identifier. If we find one, instead of emitting code for a variable\naccess, we compile the assigned value and then emit an assignment instruction.\n\nThat's the last instruction we need to add in this chapter.\n\n^code set-global-op (1 before, 1 after)\n\nAs you'd expect, its runtime behavior is similar to defining a new variable.\n\n^code interpret-set-global (1 before, 1 after)\n\nThe main difference is what happens when the key doesn't already exist in the\nglobals hash table. If the variable hasn't been defined yet, it's a runtime\nerror to try to assign to it. Lox [doesn't do implicit variable\ndeclaration][implicit].\n\n\n\nThe other difference is that setting a variable doesn't pop the value off the\nstack. Remember, assignment is an expression, so it needs to leave that value\nthere in case the assignment is nested inside some larger expression.\n\n[implicit]: statements-and-state.html#design-note\n\nAdd a dash of disassembly:\n\n^code disassemble-set-global (2 before, 1 after)\n\nSo we're done, right? Well... not quite. We've made a mistake! Take a gander at:\n\n```lox\na * b = c + d;\n```\n\nAccording to Lox's grammar, `=` has the lowest precedence, so this should be\nparsed roughly like:\n\n\"The\n\nObviously, `a * b` isn't a valid assignment target, so\nthis should be a syntax error. But here's what our parser does:\n\n\n\n1. First, `parsePrecedence()` parses `a` using the `variable()` prefix parser.\n1. After that, it enters the infix parsing loop.\n1. It reaches the `*` and calls `binary()`.\n1. That recursively calls `parsePrecedence()` to parse the right-hand operand.\n1. That calls `variable()` again for parsing `b`.\n1. Inside that call to `variable()`, it looks for a trailing `=`. It sees one\n and thus parses the rest of the line as an assignment.\n\nIn other words, the parser sees the above code like:\n\n\"The\n\nWe've messed up the precedence handling because `variable()` doesn't take into\naccount the precedence of the surrounding expression that contains the variable.\nIf the variable happens to be the right-hand side of an infix operator, or the\noperand of a unary operator, then that containing expression is too high\nprecedence to permit the `=`.\n\nTo fix this, `variable()` should look for and consume the `=` only if it's in\nthe context of a low-precedence expression. The code that knows the current\nprecedence is, logically enough, `parsePrecedence()`. The `variable()` function\ndoesn't need to know the actual level. It just cares that the precedence is low\nenough to allow assignment, so we pass that fact in as a Boolean.\n\n^code prefix-rule (4 before, 2 after)\n\nSince assignment is the lowest-precedence expression, the only time we allow an\nassignment is when parsing an assignment expression or top-level expression like\nin an expression statement. That flag makes its way to the parser function here:\n\n^code variable\n\nWhich passes it through a new parameter:\n\n^code named-variable-signature (1 after)\n\nAnd then finally uses it here:\n\n^code named-variable-can-assign (2 before, 1 after)\n\nThat's a lot of plumbing to get literally one bit of data to the right place in\nthe compiler, but arrived it has. If the variable is nested inside some\nexpression with higher precedence, `canAssign` will be `false` and this will\nignore the `=` even if there is one there. Then `namedVariable()` returns, and\nexecution eventually makes its way back to `parsePrecedence()`.\n\nThen what? What does the compiler do with our broken example from before? Right\nnow, `variable()` won't consume the `=`, so that will be the current token. The\ncompiler returns back to `parsePrecedence()` from the `variable()` prefix parser\nand then tries to enter the infix parsing loop. There is no parsing function\nassociated with `=`, so it skips that loop.\n\nThen `parsePrecedence()` silently returns back to the caller. That also isn't\nright. If the `=` doesn't get consumed as part of the expression, nothing else\nis going to consume it. It's an error and we should report it.\n\n^code invalid-assign (2 before, 1 after)\n\nWith that, the previous bad program correctly gets an error at compile time. OK,\n*now* are we done? Still not quite. See, we're passing an argument to one of the\nparse functions. But those functions are stored in a table of function pointers,\nso all of the parse functions need to have the same type. Even though most parse\nfunctions don't support being used as an assignment target -- setters are the\nonly other one -- our friendly C compiler requires\nthem *all* to accept the parameter.\n\n\n\nSo we're going to finish off this chapter with some grunt work. First, let's go\nahead and pass the flag to the infix parse functions.\n\n^code infix-rule (1 before, 1 after)\n\nWe'll need that for setters eventually. Then we'll fix the typedef for the\nfunction type.\n\n^code parse-fn-type (2 before, 2 after)\n\nAnd some completely tedious code to accept this parameter in all of our existing\nparse functions. Here:\n\n^code binary (1 after)\n\nAnd here:\n\n^code parse-literal (1 after)\n\nAnd here:\n\n^code grouping (1 after)\n\nAnd here:\n\n^code number (1 after)\n\nAnd here too:\n\n^code string (1 after)\n\nAnd, finally:\n\n^code unary (1 after)\n\nPhew! We're back to a C program we can compile. Fire it up and now you can run\nthis:\n\n```lox\nvar breakfast = \"beignets\";\nvar beverage = \"cafe au lait\";\nbreakfast = \"beignets with \" + beverage;\n\nprint breakfast;\n```\n\nIt's starting to look like real code for an actual language!\n\n
\n\n## Challenges\n\n1. The compiler adds a global variable's name to the constant table as a string\n every time an identifier is encountered. It creates a new constant each\n time, even if that variable name is already in a previous slot in the\n constant table. That's wasteful in cases where the same variable is\n referenced multiple times by the same function. That, in turn, increases the\n odds of filling up the constant table and running out of slots since we\n allow only 256 constants in a single chunk.\n\n Optimize this. How does your optimization affect the performance of the\n compiler compared to the runtime? Is this the right trade-off?\n\n2. Looking up a global variable by name in a hash table each time it is used\n is pretty slow, even with a good hash table. Can you come up with a more\n efficient way to store and access global variables without changing the\n semantics?\n\n3. When running in the REPL, a user might write a function that references an\n unknown global variable. Then, in the next line, they declare the variable.\n Lox should handle this gracefully by not reporting an \"unknown variable\"\n compile error when the function is first defined.\n\n But when a user runs a Lox *script*, the compiler has access to the full\n text of the entire program before any code is run. Consider this program:\n\n ```lox\n fun useVar() {\n print oops;\n }\n\n var ooops = \"too many o's!\";\n ```\n\n Here, we can tell statically that `oops` will not be defined because there\n is *no* declaration of that global anywhere in the program. Note that\n `useVar()` is never called either, so even though the variable isn't\n defined, no runtime error will occur because it's never used either.\n\n We could report mistakes like this as compile errors, at least when running\n from a script. Do you think we should? Justify your answer. What do other\n scripting languages you know do?\n\n
\n" }, { "path": "book/hash-tables.md", "content": "> Hash, x. There is no definition for this word -- nobody knows what hash is.\n>\n> Ambrose Bierce, The Unabridged Devil's Dictionary\n\nBefore we can add variables to our burgeoning virtual machine, we need some way\nto look up a value given a variable's name. Later, when we add classes, we'll\nalso need a way to store fields on instances. The perfect data structure for\nthese problems and others is a hash table.\n\nYou probably already know what a hash table is, even if you don't know it by\nthat name. If you're a Java programmer, you call them \"HashMaps\". C# and Python\nusers call them \"dictionaries\". In C++, it's an \"unordered map\". \"Objects\" in\nJavaScript and \"tables\" in Lua are hash tables under the hood, which is what\ngives them their flexibility.\n\nA hash table, whatever your language calls it, associates a set of **keys** with\na set of **values**. Each key/value pair is an **entry** in the table. Given a\nkey, you can look up its corresponding value. You can add new key/value pairs\nand remove entries by key. If you add a new value for an existing key, it\nreplaces the previous entry.\n\nHash tables appear in so many languages because they are incredibly powerful.\nMuch of this power comes from one metric: given a key, a hash table returns the\ncorresponding value in constant time, *regardless\nof how many keys are in the hash table*.\n\n\n\nThat's pretty remarkable when you think about it. Imagine you've got a big stack\nof business cards and I ask you to find a certain person. The bigger the pile\nis, the longer it will take. Even if the pile is nicely sorted and you've got\nthe manual dexterity to do a binary search by hand, you're still talking\n*O(log n)*. But with a hash table, it takes the\nsame time to find that business card when the stack has ten cards as when it has\na million.\n\n\n\n## An Array of Buckets\n\nA complete, fast hash table has a couple of moving parts. I'll introduce them\none at a time by working through a couple of toy problems and their solutions.\nEventually, we'll build up to a data structure that can associate any set of\nnames with their values.\n\nFor now, imagine if Lox was a *lot* more restricted in variable names. What if a\nvariable's name could only be a single lowercase\nletter. How could we very efficiently represent a set of variable names and\ntheir values?\n\n\n\nWith only 26 possible variables (27 if you consider underscore a \"letter\", I\nguess), the answer is easy. Declare a fixed-size array with 26 elements. We'll\nfollow tradition and call each element a **bucket**. Each represents a variable\nwith `a` starting at index zero. If there's a value in the array at some\nletter's index, then that key is present with that value. Otherwise, the bucket\nis empty and that key/value pair isn't in the data structure.\n\n\n\nMemory usage is great -- just a single, reasonably sized array. There's some waste from the empty buckets, but it's\nnot huge. There's no overhead for node pointers, padding, or other stuff you'd\nget with something like a linked list or tree.\n\nPerformance is even better. Given a variable name -- its character -- you can\nsubtract the ASCII value of `a` and use the result to index directly into the\narray. Then you can either look up the existing value or store a new value\ndirectly into that slot. It doesn't get much faster than that.\n\nThis is sort of our Platonic ideal data structure. Lightning fast, dead simple,\nand compact in memory. As we add support for more complex keys, we'll have to\nmake some concessions, but this is what we're aiming for. Even once you add in\nhash functions, dynamic resizing, and collision resolution, this is still the\ncore of every hash table out there -- a contiguous array of buckets that you\nindex directly into.\n\n### Load factor and wrapped keys\n\nConfining Lox to single-letter variables would make our job as implementers\neasier, but it's probably no fun programming in a language that gives you only\n26 storage locations. What if we loosened it a little and allowed variables up\nto eight characters long?\n\n\n\nThat's small enough that we can pack all eight characters into a 64-bit integer\nand easily turn the string into a number. We can then use it as an array index.\nOr, at least, we could if we could somehow allocate a 295,148 *petabyte* array.\nMemory's gotten cheaper over time, but not quite *that* cheap. Even if we could\nmake an array that big, it would be heinously wasteful. Almost every bucket\nwould be empty unless users started writing way bigger Lox programs than we've\nanticipated.\n\nEven though our variable keys cover the full 64-bit numeric range, we clearly\ndon't need an array that large. Instead, we allocate an array with more than\nenough capacity for the entries we need, but not unreasonably large. We map the\nfull 64-bit keys down to that smaller range by taking the value modulo the size\nof the array. Doing that essentially folds the larger numeric range onto itself\nuntil it fits the smaller range of array elements.\n\nFor example, say we want to store \"bagel\". We allocate an array with eight\nelements, plenty enough to store it and more later. We treat the key string as a\n64-bit integer. On a little-endian machine like Intel, packing those characters\ninto a 64-bit word puts the first letter, \"b\" (ASCII value 98), in the\nleast-significant byte. We take that integer modulo the array size (8) to fit it in the bounds and get a bucket index, 2.\nThen we store the value there as usual.\n\n\n\nUsing the array size as a modulus lets us map the key's numeric range down to\nfit an array of any size. We can thus control the number of buckets\nindependently of the key range. That solves our waste problem, but introduces a\nnew one. Any two variables whose key number has the same remainder when divided\nby the array size will end up in the same bucket. Keys can **collide**. For\nexample, if we try to add \"jam\", it also ends up in bucket 2.\n\n\"'Bagel'\n\nWe have some control over this by tuning the array size. The bigger the array,\nthe fewer the indexes that get mapped to the same bucket and the fewer the\ncollisions that are likely to occur. Hash table implementers track this\ncollision likelihood by measuring the table's **load factor**. It's defined as\nthe number of entries divided by the number of buckets. So a hash table with\nfive entries and an array of 16 elements has a load factor of 0.3125. The higher\nthe load factor, the greater the chance of collisions.\n\nOne way we mitigate collisions is by resizing the array. Just like the dynamic\narrays we implemented earlier, we reallocate and grow the hash table's array as\nit fills up. Unlike a regular dynamic array, though, we won't wait until the\narray is *full*. Instead, we pick a desired load factor and grow the array when\nit goes over that.\n\n## Collision Resolution\n\nEven with a very low load factor, collisions can still occur. The [*birthday\nparadox*][birthday] tells us that as the number of entries in the hash table\nincreases, the chance of collision increases very quickly. We can pick a large\narray size to reduce that, but it's a losing game. Say we wanted to store a\nhundred items in a hash table. To keep the chance of collision below a\nstill-pretty-high 10%, we need an array with at least 47,015 elements. To get\nthe chance below 1% requires an array with 492,555 elements, over 4,000 empty\nbuckets for each one in use.\n\n[birthday]: https://en.wikipedia.org/wiki/Birthday_problem\n\nA low load factor can make collisions rarer, but the\n[*pigeonhole principle*][pigeon] tells us we can never eliminate them entirely.\nIf you've got five pet pigeons and four holes to put them in, at least one hole\nis going to end up with more than one pigeon. With 18,446,744,073,709,551,616\ndifferent variable names, any reasonably sized array can potentially end up with\nmultiple keys in the same bucket.\n\n[pigeon]: https://en.wikipedia.org/wiki/Pigeonhole_principle\n\nThus we still have to handle collisions gracefully when they occur. Users don't\nlike it when their programming language can look up variables correctly only\n*most* of the time.\n\n\n\n### Separate chaining\n\nTechniques for resolving collisions fall into two broad categories. The first is\n**separate chaining**. Instead of each bucket containing a single entry, we let\nit contain a collection of them. In the classic implementation, each bucket\npoints to a linked list of entries. To look up an entry, you find its bucket and\nthen walk the list until you find an entry with the matching key.\n\n\"An\n\nIn catastrophically bad cases where every entry collides in the same bucket, the\ndata structure degrades into a single unsorted linked list with *O(n)* lookup.\nIn practice, it's easy to avoid that by controlling the load factor and how\nentries get scattered across buckets. In typical separate-chained hash tables,\nit's rare for a bucket to have more than one or two entries.\n\nSeparate chaining is conceptually simple -- it's literally an array of linked\nlists. Most operations are straightforward to implement, even deletion which, as\nwe'll see, can be a pain. But it's not a great fit for modern CPUs. It has a lot\nof overhead from pointers and tends to scatter little linked list nodes around in memory which isn't great for cache usage.\n\n\n\n### Open addressing\n\nThe other technique is called **open addressing** or\n(confusingly) **closed hashing**. With this technique, all entries live directly\nin the bucket array, with one entry per bucket. If two entries collide in the\nsame bucket, we find a different empty bucket to use instead.\n\n\n\nStoring all entries in a single, big, contiguous array is great for keeping the\nmemory representation simple and fast. But it makes all of the operations on the\nhash table more complex. When inserting an entry, its bucket may be full,\nsending us to look at another bucket. That bucket itself may be occupied and so\non. This process of finding an available bucket is called **probing**, and the\norder that you examine buckets is a **probe sequence**.\n\nThere are a number of algorithms for determining\nwhich buckets to probe and how to decide which entry goes in which bucket.\nThere's been a ton of research here because even slight tweaks can have a large\nperformance impact. And, on a data structure as heavily used as hash tables,\nthat performance impact touches a very large number of real-world programs\nacross a range of hardware capabilities.\n\n\n\nAs usual in this book, we'll pick the simplest one that gets the job done\nefficiently. That's good old **linear probing**. When looking for an entry, we\nlook in the first bucket its key maps to. If it's not in there, we look in the\nvery next element in the array, and so on. If we reach the end, we wrap back\naround to the beginning.\n\nThe good thing about linear probing is that it's cache friendly. Since you walk\nthe array directly in memory order, it keeps the CPU's cache lines full and\nhappy. The bad thing is that it's prone to **clustering**. If you have a lot of\nentries with numerically similar key values, you can end up with a lot of\ncolliding, overflowing buckets right next to each other.\n\nCompared to separate chaining, open addressing can be harder to wrap your head\naround. I think of open addressing as similar to separate chaining except that\nthe \"list\" of nodes is threaded through the bucket array itself. Instead of\nstoring the links between them in pointers, the connections are calculated\nimplicitly by the order that you look through the buckets.\n\nThe tricky part is that more than one of these implicit lists may be interleaved\ntogether. Let's walk through an example that covers all the interesting cases.\nWe'll ignore values for now and just worry about a set of keys. We start with an\nempty array of 8 buckets.\n\n\"An\n\nWe decide to insert \"bagel\". The first letter, \"b\" (ASCII value 98), modulo the\narray size (8) puts it in bucket 2.\n\n\"Bagel\n\nNext, we insert \"jam\". That also wants to go in bucket 2 (106 mod 8 = 2), but\nthat bucket's taken. We keep probing to the next bucket. It's empty, so we put\nit there.\n\n\"Jam\n\nWe insert \"fruit\", which happily lands in bucket 6.\n\n\"Fruit\n\nLikewise, \"migas\" can go in its preferred bucket 5.\n\n\"Migas\n\nWhen we try to insert \"eggs\", it also wants to be in bucket 5. That's full, so we\nskip to 6. Bucket 6 is also full. Note that the entry in there is *not* part of\nthe same probe sequence. \"Fruit\" is in its preferred bucket, 6. So the 5 and 6\nsequences have collided and are interleaved. We skip over that and finally put\n\"eggs\" in bucket 7.\n\n\"Eggs\n\nWe run into a similar problem with \"nuts\". It can't land in 6 like it wants to.\nNor can it go into 7. So we keep going. But we've reached the end of the array,\nso we wrap back around to 0 and put it there.\n\n\"Nuts\n\nIn practice, the interleaving turns out to not be much of a problem. Even in\nseparate chaining, we need to walk the list to check each entry's key because\nmultiple keys can reduce to the same bucket. With open addressing, we need to do\nthat same check, and that also covers the case where you are stepping over\nentries that \"belong\" to a different original bucket.\n\n## Hash Functions\n\nWe can now build ourselves a reasonably efficient table for storing variable\nnames up to eight characters long, but that limitation is still annoying. In\norder to relax the last constraint, we need a way to take a string of any length\nand convert it to a fixed-size integer.\n\nFinally, we get to the \"hash\" part of \"hash table\". A **hash function** takes\nsome larger blob of data and \"hashes\" it to produce a fixed-size integer **hash\ncode** whose value depends on all of the bits of the original data. A good hash function has three main goals:\n\n\n\n* **It must be *deterministic*.** The same input must always hash to the same\n number. If the same variable ends up in different buckets at different\n points in time, it's gonna get really hard to find it.\n\n* **It must be *uniform*.** Given a typical set of inputs, it should produce a\n wide and evenly distributed range of output numbers, with as few clumps or\n patterns as possible. We want it to scatter\n values across the whole numeric range to minimize collisions and clustering.\n\n* **It must be *fast*.** Every operation on the hash table requires us to hash\n the key first. If hashing is slow, it can potentially cancel out the speed\n of the underlying array storage.\n\n\n\nThere is a veritable pile of hash functions out there. Some are old and\noptimized for architectures no one uses anymore. Some are designed to be fast,\nothers cryptographically secure. Some take advantage of vector instructions and\ncache sizes for specific chips, others aim to maximize portability.\n\nThere are people out there for whom designing and evaluating hash functions is,\nlike, their *jam*. I admire them, but I'm not mathematically astute enough to\n*be* one. So for clox, I picked a simple, well-worn hash function called\n[FNV-1a][] that's served me fine over the years. Consider trying out different ones in your code and see if they make\na difference.\n\n[fnv-1a]: http://www.isthe.com/chongo/tech/comp/fnv/\n\n\n\nOK, that's a quick run through of buckets, load factors, open addressing,\ncollision resolution, and hash functions. That's an awful lot of text and not a\nlot of real code. Don't worry if it still seems vague. Once we're done coding it\nup, it will all click into place.\n\n## Building a Hash Table\n\nThe great thing about hash tables compared to other classic techniques like\nbalanced search trees is that the actual data structure is so simple. Ours goes\ninto a new module.\n\n^code table-h\n\nA hash table is an array of entries. As in our dynamic array earlier, we keep\ntrack of both the allocated size of the array (`capacity`) and the number of\nkey/value pairs currently stored in it (`count`). The ratio of count to capacity\nis exactly the load factor of the hash table.\n\nEach entry is one of these:\n\n^code entry (1 before, 2 after)\n\nIt's a simple key/value pair. Since the key is always a string, we store the ObjString pointer directly instead of\nwrapping it in a Value. It's a little faster and smaller this way.\n\n\n\nTo create a new, empty hash table, we declare a constructor-like function.\n\n^code init-table-h (2 before, 2 after)\n\nWe need a new implementation file to define that. While we're at it, let's get\nall of the pesky includes out of the way.\n\n^code table-c\n\nAs in our dynamic value array type, a hash table initially starts with zero\ncapacity and a `NULL` array. We don't allocate anything until needed. Assuming\nwe do eventually allocate something, we need to be able to free it too.\n\n^code free-table-h (1 before, 2 after)\n\nAnd its glorious implementation:\n\n^code free-table\n\nAgain, it looks just like a dynamic array. In fact, you can think of a hash\ntable as basically a dynamic array with a really strange policy for inserting\nitems. We don't need to check for `NULL` here since `FREE_ARRAY()` already\nhandles that gracefully.\n\n### Hashing strings\n\nBefore we can start putting entries in the table, we need to, well, hash them.\nTo ensure that the entries get distributed uniformly throughout the array, we\nwant a good hash function that looks at all of the bits of the key string. If it\nlooked at, say, only the first few characters, then a series of strings that all\nshared the same prefix would end up colliding in the same bucket.\n\nOn the other hand, walking the entire string to calculate the hash is kind of\nslow. We'd lose some of the performance benefit of the hash table if we had to\nwalk the string every time we looked for a key in the table. So we'll do the\nobvious thing: cache it.\n\nOver in the \"object\" module in ObjString, we add:\n\n^code obj-string-hash (1 before, 1 after)\n\nEach ObjString stores the hash code for its string. Since strings are immutable\nin Lox, we can calculate the hash code once up front and be certain that it will\nnever get invalidated. Caching it eagerly makes a kind of sense: allocating the\nstring and copying its characters over is already an *O(n)* operation, so it's a\ngood time to also do the *O(n)* calculation of the string's hash.\n\nWhenever we call the internal function to allocate a string, we pass in its\nhash code.\n\n^code allocate-string (1 after)\n\nThat function simply stores the hash in the struct.\n\n^code allocate-store-hash (1 before, 2 after)\n\nThe fun happens over at the callers. `allocateString()` is called from two\nplaces: the function that copies a string and the one that takes ownership of an\nexisting dynamically allocated string. We'll start with the first.\n\n^code copy-string-hash (1 before, 1 after)\n\nNo magic here. We calculate the hash code and then pass it along.\n\n^code copy-string-allocate (2 before, 1 after)\n\nThe other string function is similar.\n\n^code take-string-hash (1 before, 1 after)\n\nThe interesting code is over here:\n\n^code hash-string\n\nThis is the actual bona fide \"hash function\" in clox. The algorithm is called\n\"FNV-1a\", and is the shortest decent hash function I know. Brevity is certainly\na virtue in a book that aims to show you every line of code.\n\nThe basic idea is pretty simple, and many hash functions follow the same\npattern. You start with some initial hash value, usually a constant with certain\ncarefully chosen mathematical properties. Then you walk the data to be hashed.\nFor each byte (or sometimes word), you mix the bits into the hash value somehow,\nand then scramble the resulting bits around some.\n\nWhat it means to \"mix\" and \"scramble\" can get pretty sophisticated. Ultimately,\nthough, the basic goal is *uniformity* -- we want the resulting hash values to\nbe as widely scattered around the numeric range as possible to avoid collisions\nand clustering.\n\n### Inserting entries\n\nNow that string objects know their hash code, we can start putting them into\nhash tables.\n\n^code table-set-h (1 before, 2 after)\n\nThis function adds the given key/value pair to the given hash table. If an entry\nfor that key is already present, the new value overwrites the old value. The\nfunction returns `true` if a new entry was added. Here's the implementation:\n\n^code table-set\n\nMost of the interesting logic is in `findEntry()` which we'll get to soon. That\nfunction's job is to take a key and figure out which bucket in the array it\nshould go in. It returns a pointer to that bucket -- the address of the Entry in\nthe array.\n\nOnce we have a bucket, inserting is straightforward. We update the hash table's\nsize, taking care to not increase the count if we overwrote the value for an\nalready-present key. Then we copy the key and value into the corresponding\nfields in the Entry.\n\nWe're missing a little something here, though. We haven't actually allocated the\nEntry array yet. Oops! Before we can insert anything, we need to make sure we\nhave an array, and that it's big enough.\n\n^code table-set-grow (1 before, 1 after)\n\nThis is similar to the code we wrote a while back for growing a dynamic array.\nIf we don't have enough capacity to insert an item, we reallocate and grow the\narray. The `GROW_CAPACITY()` macro takes an existing capacity and grows it by\na multiple to ensure that we get amortized constant performance over a series\nof inserts.\n\nThe interesting difference here is that `TABLE_MAX_LOAD` constant.\n\n^code max-load (2 before, 1 after)\n\nThis is how we manage the table's load factor. We don't\ngrow when the capacity is completely full. Instead, we grow the array before\nthen, when the array becomes at least 75% full.\n\n\n\nWe'll get to the implementation of `adjustCapacity()` soon. First, let's look\nat that `findEntry()` function you've been wondering about.\n\n^code find-entry\n\nThis function is the real core of the hash table. It's responsible for taking a\nkey and an array of buckets, and figuring out which bucket the entry belongs in.\nThis function is also where linear probing and collision handling come into\nplay. We'll use `findEntry()` both to look up existing entries in the hash\ntable and to decide where to insert new ones.\n\nFor all that, there isn't much to it. First, we use modulo to map the key's hash\ncode to an index within the array's bounds. That gives us a bucket index where,\nideally, we'll be able to find or place the entry.\n\nThere are a few cases to check for:\n\n* If the key for the Entry at that array index is `NULL`, then the bucket is\n empty. If we're using `findEntry()` to look up something in the hash table,\n this means it isn't there. If we're using it to insert, it means we've found\n a place to add the new entry.\n\n* If the key in the bucket is equal to the key we're\n looking for, then that key is already present in the table. If we're doing a\n lookup, that's good -- we've found the key we seek. If we're doing an insert,\n this means we'll be replacing the value for that key instead of adding a new\n entry.\n\n\n\n* Otherwise, the bucket has an entry in it, but with a different key. This is\n a collision. In that case, we start probing. That's what that `for` loop\n does. We start at the bucket where the entry would ideally go. If that\n bucket is empty or has the same key, we're done. Otherwise, we advance to\n the next element -- this is the *linear* part of \"linear probing\" -- and\n check there. If we go past the end of the array, that second modulo operator\n wraps us back around to the beginning.\n\nWe exit the loop when we find either an empty bucket or a bucket with the same\nkey as the one we're looking for. You might be wondering about an infinite loop.\nWhat if we collide with *every* bucket? Fortunately, that can't happen thanks to\nour load factor. Because we grow the array as soon as it gets close to being\nfull, we know there will always be empty buckets.\n\nWe return directly from within the loop, yielding a pointer to the found Entry\nso the caller can either insert something into it or read from it. Way back in\n`tableSet()`, the function that first kicked this off, we store the new entry in\nthat returned bucket and we're done.\n\n### Allocating and resizing\n\nBefore we can put entries in the hash table, we do need a place to actually\nstore them. We need to allocate an array of buckets. That happens in this\nfunction:\n\n^code table-adjust-capacity\n\nWe create a bucket array with `capacity` entries. After we allocate the array,\nwe initialize every element to be an empty bucket and then store the array (and\nits capacity) in the hash table's main struct. This code is fine for when we\ninsert the very first entry into the table, and we require the first allocation\nof the array. But what about when we already have one and we need to grow it?\n\nBack when we were doing a dynamic array, we could just use `realloc()` and let\nthe C standard library copy everything over. That doesn't work for a hash table.\nRemember that to choose the bucket for each entry, we take its hash key *modulo\nthe array size*. That means that when the array size changes, entries may end up\nin different buckets.\n\nThose new buckets may have new collisions that we need to deal with. So the\nsimplest way to get every entry where it belongs is to rebuild the table from\nscratch by re-inserting every entry into the new empty array.\n\n^code re-hash (2 before, 2 after)\n\nWe walk through the old array front to back. Any time we find a non-empty\nbucket, we insert that entry into the new array. We use `findEntry()`, passing\nin the *new* array instead of the one currently stored in the Table. (This is\nwhy `findEntry()` takes a pointer directly to an Entry array and not the whole\n`Table` struct. That way, we can pass the new array and capacity before we've\nstored those in the struct.)\n\nAfter that's done, we can release the memory for the old array.\n\n^code free-old-array (3 before, 1 after)\n\nWith that, we have a hash table that we can stuff as many entries into as we\nlike. It handles overwriting existing keys and growing itself as needed to\nmaintain the desired load capacity.\n\nWhile we're at it, let's also define a helper function for copying all of the\nentries of one hash table into another.\n\n^code table-add-all-h (1 before, 2 after)\n\nWe won't need this until much later when we support method inheritance, but we\nmay as well implement it now while we've got all the hash table stuff fresh in\nour minds.\n\n^code table-add-all\n\nThere's not much to say about this. It walks the bucket array of the source hash\ntable. Whenever it finds a non-empty bucket, it adds the entry to the\ndestination hash table using the `tableSet()` function we recently defined.\n\n### Retrieving values\n\nNow that our hash table contains some stuff, let's start pulling things back\nout. Given a key, we can look up the corresponding value, if there is one, with\nthis function:\n\n^code table-get-h (1 before, 1 after)\n\nYou pass in a table and a key. If it finds an entry with that key, it returns\n`true`, otherwise it returns `false`. If the entry exists, the `value` output\nparameter points to the resulting value.\n\nSince `findEntry()` already does the hard work, the implementation isn't bad.\n\n^code table-get\n\nIf the table is completely empty, we definitely won't find the entry, so we\ncheck for that first. This isn't just an optimization -- it also ensures that we\ndon't try to access the bucket array when the array is `NULL`. Otherwise, we let\n`findEntry()` work its magic. That returns a pointer to a bucket. If the bucket\nis empty, which we detect by seeing if the key is `NULL`, then we didn't find an\nEntry with our key. If `findEntry()` does return a non-empty Entry, then that's\nour match. We take the Entry's value and copy it to the output parameter so the\ncaller can get it. Piece of cake.\n\n### Deleting entries\n\nThere is one more fundamental operation a full-featured hash table needs to\nsupport: removing an entry. This seems pretty obvious, if you can add things,\nyou should be able to *un*-add them, right? But you'd be surprised how many\ntutorials on hash tables omit this.\n\nI could have taken that route too. In fact, we use deletion in clox only in a\ntiny edge case in the VM. But if you want to actually understand how to\ncompletely implement a hash table, this feels important. I can sympathize with\ntheir desire to overlook it. As we'll see, deleting from a hash table that uses\nopen addressing is tricky.\n\n\n\nAt least the declaration is simple.\n\n^code table-delete-h (1 before, 1 after)\n\nThe obvious approach is to mirror insertion. Use `findEntry()` to look up the\nentry's bucket. Then clear out the bucket. Done!\n\nIn cases where there are no collisions, that works fine. But if a collision has\noccurred, then the bucket where the entry lives may be part of one or more\nimplicit probe sequences. For example, here's a hash table containing three keys\nall with the same preferred bucket, 2:\n\n\"A\n\nRemember that when we're walking a probe sequence to find an entry, we know\nwe've reached the end of a sequence and that the entry isn't present when we hit\nan empty bucket. It's like the probe sequence is a list of entries and an empty\nentry terminates that list.\n\nIf we delete \"biscuit\" by simply clearing the Entry, then we break that probe\nsequence in the middle, leaving the trailing entries orphaned and unreachable.\nSort of like removing a node from a linked list without relinking the pointer\nfrom the previous node to the next one.\n\nIf we later try to look for \"jam\", we'd start at \"bagel\", stop at the next\nempty Entry, and never find it.\n\n\"The\n\nTo solve this, most implementations use a trick called **tombstones**. Instead of clearing the entry on\ndeletion, we replace it with a special sentinel entry called a \"tombstone\". When\nwe are following a probe sequence during a lookup, and we hit a tombstone, we\n*don't* treat it like an empty slot and stop iterating. Instead, we keep going\nso that deleting an entry doesn't break any implicit collision chains and we can\nstill find entries after it.\n\n\"Instead\n\nThe code looks like this:\n\n^code table-delete\n\nFirst, we find the bucket containing the entry we want to delete. (If we don't\nfind it, there's nothing to delete, so we bail out.) We replace the entry with a\ntombstone. In clox, we use a `NULL` key and a `true` value to represent that,\nbut any representation that can't be confused with an empty bucket or a valid\nentry works.\n\n\n\nThat's all we need to do to delete an entry. Simple and fast. But all of the\nother operations need to correctly handle tombstones too. A tombstone is a sort\nof \"half\" entry. It has some of the characteristics of a present entry, and some\nof the characteristics of an empty one.\n\nWhen we are following a probe sequence during a lookup, and we hit a tombstone,\nwe note it and keep going.\n\n^code find-tombstone (2 before, 2 after)\n\nThe first time we pass a tombstone, we store it in this local variable:\n\n^code find-entry-tombstone (1 before, 1 after)\n\nIf we reach a truly empty entry, then the key isn't present. In that case, if we\nhave passed a tombstone, we return its bucket instead of the later empty one. If\nwe're calling `findEntry()` in order to insert a node, that lets us treat the\ntombstone bucket as empty and reuse it for the new entry.\n\nReusing tombstone slots automatically like this helps reduce the number of\ntombstones wasting space in the bucket array. In typical use cases where there\nis a mixture of insertions and deletions, the number of tombstones grows for a\nwhile and then tends to stabilize.\n\nEven so, there's no guarantee that a large number of deletes won't cause the\narray to be full of tombstones. In the very worst case, we could end up with\n*no* empty buckets. That would be bad because, remember, the only thing\npreventing an infinite loop in `findEntry()` is the assumption that we'll\neventually hit an empty bucket.\n\nSo we need to be thoughtful about how tombstones interact with the table's load\nfactor and resizing. The key question is, when calculating the load factor,\nshould we treat tombstones like full buckets or empty ones?\n\n### Counting tombstones\n\nIf we treat tombstones like full buckets, then we may end up with a bigger array\nthan we probably need because it artificially inflates the load factor. There\nare tombstones we could reuse, but they aren't treated as unused so we end up\ngrowing the array prematurely.\n\nBut if we treat tombstones like empty buckets and *don't* include them in the\nload factor, then we run the risk of ending up with *no* actual empty buckets to\nterminate a lookup. An infinite loop is a much worse problem than a few extra\narray slots, so for load factor, we consider tombstones to be full buckets.\n\nThat's why we don't reduce the count when deleting an entry in the previous\ncode. The count is no longer the number of entries in the hash table, it's the\nnumber of entries plus tombstones. That implies that we increment the count\nduring insertion only if the new entry goes into an entirely empty bucket.\n\n^code set-increment-count (1 before, 2 after)\n\nIf we are replacing a tombstone with a new entry, the bucket has already been\naccounted for and the count doesn't change.\n\nWhen we resize the array, we allocate a new array and re-insert all of the\nexisting entries into it. During that process, we *don't* copy the tombstones\nover. They don't add any value since we're rebuilding the probe sequences\nanyway, and would just slow down lookups. That means we need to recalculate the\ncount since it may change during a resize. So we clear it out:\n\n^code resize-init-count (2 before, 1 after)\n\nThen each time we find a non-tombstone entry, we increment it.\n\n^code resize-increment-count (1 before, 1 after)\n\nThis means that when we grow the capacity, we may end up with *fewer* entries in\nthe resulting larger array because all of the tombstones get discarded. That's a\nlittle wasteful, but not a huge practical problem.\n\nI find it interesting that much of the work to support deleting entries is in\n`findEntry()` and `adjustCapacity()`. The actual delete logic is quite simple\nand fast. In practice, deletions tend to be rare, so you'd expect a hash table\nto do as much work as it can in the delete function and leave the other\nfunctions alone to keep them faster. With our tombstone approach, deletes are\nfast, but lookups get penalized.\n\nI did a little benchmarking to test this out in a few different deletion\nscenarios. I was surprised to discover that tombstones did end up being faster\noverall compared to doing all the work during deletion to reinsert the affected\nentries.\n\nBut if you think about it, it's not that the tombstone approach pushes the work\nof fully deleting an entry to other operations, it's more that it makes deleting\n*lazy*. At first, it does the minimal work to turn the entry into a tombstone.\nThat can cause a penalty when later lookups have to skip over it. But it also\nallows that tombstone bucket to be reused by a later insert too. That reuse is a\nvery efficient way to avoid the cost of rearranging all of the following\naffected entries. You basically recycle a node in the chain of probed entries.\nIt's a neat trick.\n\n## String Interning\n\nWe've got ourselves a hash table that mostly works, though it has a critical\nflaw in its center. Also, we aren't using it for anything yet. It's time to\naddress both of those and, in the process, learn a classic technique used by\ninterpreters.\n\nThe reason the hash table doesn't totally work is that when `findEntry()` checks\nto see if an existing key matches the one it's looking for, it uses `==` to\ncompare two strings for equality. That only returns true if the two keys are the\nexact same string in memory. Two separate strings with the same characters\nshould be considered equal, but aren't.\n\nRemember, back when we added strings in the last chapter, we added [explicit\nsupport to compare the strings character-by-character][equals] in order to get\ntrue value equality. We could do that in `findEntry()`, but that's slow.\n\n[equals]: strings.html#operations-on-strings\n\n\n\nInstead, we'll use a technique called **string interning**. The core problem is\nthat it's possible to have different strings in memory with the same characters.\nThose need to behave like equivalent values even though they are distinct\nobjects. They're essentially duplicates, and we have to compare all of their\nbytes to detect that.\n\nString interning is a process of deduplication. We\ncreate a collection of \"interned\" strings. Any string in that collection is\nguaranteed to be textually distinct from all others. When you intern a string,\nyou look for a matching string in the collection. If found, you use that\noriginal one. Otherwise, the string you have is unique, so you add it to the\ncollection.\n\n\n\nIn this way, you know that each sequence of characters is represented by only\none string in memory. This makes value equality trivial. If two strings point\nto the same address in memory, they are obviously the same string and must be\nequal. And, because we know strings are unique, if two strings point to\ndifferent addresses, they must be distinct strings.\n\nThus, pointer equality exactly matches value equality. Which in turn means that\nour existing `==` in `findEntry()` does the right thing. Or, at least, it will\nonce we intern all the strings. In order to reliably deduplicate all strings,\nthe VM needs to be able to find every string that's created. We do that by\ngiving it a hash table to store them all.\n\n^code vm-strings (1 before, 1 after)\n\nAs usual, we need an include.\n\n^code vm-include-table (1 before, 1 after)\n\nWhen we spin up a new VM, the string table is empty.\n\n^code init-strings (1 before, 1 after)\n\nAnd when we shut down the VM, we clean up any resources used by the table.\n\n^code free-strings (1 before, 1 after)\n\nSome languages have a separate type or an explicit step to intern a string. For\nclox, we'll automatically intern every one. That means whenever we create a new\nunique string, we add it to the table.\n\n^code allocate-store-string (1 before, 1 after)\n\nWe're using the table more like a hash *set* than a hash *table*. The keys are\nthe strings and those are all we care about, so we just use `nil` for the\nvalues.\n\nThis gets a string into the table assuming that it's unique, but we need to\nactually check for duplication before we get here. We do that in the two\nhigher-level functions that call `allocateString()`. Here's one:\n\n^code copy-string-intern (1 before, 1 after)\n\nWhen copying a string into a new LoxString, we look it up in the string table\nfirst. If we find it, instead of \"copying\", we just return a reference to that\nstring. Otherwise, we fall through, allocate a new string, and store it in the\nstring table.\n\nTaking ownership of a string is a little different.\n\n^code take-string-intern (1 before, 1 after)\n\nAgain, we look up the string in the string table first. If we find it, before we\nreturn it, we free the memory for the string that was passed in. Since ownership\nis being passed to this function and we no longer need the duplicate string,\nit's up to us to free it.\n\nBefore we get to the new function we need to write, there's one more include.\n\n^code object-include-table (1 before, 1 after)\n\nTo look for a string in the table, we can't use the normal `tableGet()` function\nbecause that calls `findEntry()`, which has the exact problem with duplicate\nstrings that we're trying to fix right now. Instead, we use this new function:\n\n^code table-find-string-h (1 before, 2 after)\n\nThe implementation looks like so:\n\n^code table-find-string\n\nIt appears we have copy-pasted `findEntry()`. There is a lot of redundancy, but\nalso a couple of key differences. First, we pass in the raw character array of\nthe key we're looking for instead of an ObjString. At the point that we call\nthis, we haven't created an ObjString yet.\n\nSecond, when checking to see if we found the key, we look at the actual strings.\nWe first see if they have matching lengths and hashes. Those are quick to check\nand if they aren't equal, the strings definitely aren't the same.\n\nIf there is a hash collision, we do an actual character-by-character string\ncomparison. This is the one place in the VM where we actually test strings for\ntextual equality. We do it here to deduplicate strings and then the rest of the\nVM can take for granted that any two strings at different addresses in memory\nmust have different contents.\n\nIn fact, now that we've interned all the strings, we can take advantage of it in\nthe bytecode interpreter. When a user does `==` on two objects that happen to be\nstrings, we don't need to test the characters any more.\n\n^code equal (1 before, 1 after)\n\nWe've added a little overhead when creating strings to intern them. But in\nreturn, at runtime, the equality operator on strings is much faster. With that,\nwe have a full-featured hash table ready for us to use for tracking variables,\ninstances, or any other key-value pairs that might show up.\n\nWe also sped up testing strings for equality. This is nice for when the user\ndoes `==` on strings. But it's even more critical in a dynamically typed\nlanguage like Lox where method calls and instance fields are looked up by name\nat runtime. If testing a string for equality is slow, then that means looking up\na method by name is slow. And if *that's* slow in your object-oriented language,\nthen *everything* is slow.\n\n
\n\n## Challenges\n\n1. In clox, we happen to only need keys that are strings, so the hash table we\n built is hardcoded for that key type. If we exposed hash tables to Lox users\n as a first-class collection, it would be useful to support different kinds\n of keys.\n\n Add support for keys of the other primitive types: numbers, Booleans, and\n `nil`. Later, clox will support user-defined classes. If we want to support\n keys that are instances of those classes, what kind of complexity does that\n add?\n\n1. Hash tables have a lot of knobs you can tweak that affect their performance.\n You decide whether to use separate chaining or open addressing. Depending on\n which fork in that road you take, you can tune how many entries are stored\n in each node, or the probing strategy you use. You control the hash\n function, load factor, and growth rate.\n\n All of this variety wasn't created just to give CS doctoral candidates\n something to publish theses on: each has its\n uses in the many varied domains and hardware scenarios where hashing comes\n into play. Look up a few hash table implementations in different open source\n systems, research the choices they made, and try to figure out why they did\n things that way.\n\n \n\n1. Benchmarking a hash table is notoriously difficult. A hash table\n implementation may perform well with some keysets and poorly with others. It\n may work well at small sizes but degrade as it grows, or vice versa. It may\n choke when deletions are common, but fly when they aren't. Creating\n benchmarks that accurately represent how your users will use the hash table\n is a challenge.\n\n Write a handful of different benchmark programs to validate our hash table\n implementation. How does the performance vary between them? Why did you\n choose the specific test cases you chose?\n\n
\n" }, { "path": "book/index.md", "content": "This text is not used. All of the content is in the index.html template.\n" }, { "path": "book/inheritance.md", "content": "> Once we were blobs in the sea, and then fishes, and then lizards and rats and\n> then monkeys, and hundreds of things in between. This hand was once a fin,\n> this hand once had claws! In my human mouth I have the pointy teeth of a wolf\n> and the chisel teeth of a rabbit and the grinding teeth of a cow! Our blood is\n> as salty as the sea we used to live in! When we're frightened, the hair on our\n> skin stands up, just like it did when we had fur. We are history! Everything\n> we've ever been on the way to becoming us, we still are.\n>\n> Terry Pratchett, A Hat Full of Sky\n\nCan you believe it? We've reached the last chapter of [Part II][]. We're almost\ndone with our first Lox interpreter. The [previous chapter][] was a big ball of\nintertwined object-orientation features. I couldn't separate those from each\nother, but I did manage to untangle one piece. In this chapter, we'll finish\noff Lox's class support by adding inheritance.\n\n[part ii]: a-tree-walk-interpreter.html\n[previous chapter]: classes.html\n\nInheritance appears in object-oriented languages all the way back to the first one, [Simula][]. Early on, Kristen Nygaard and\nOle-Johan Dahl noticed commonalities across classes in the simulation programs\nthey wrote. Inheritance gave them a way to reuse the code for those similar\nparts.\n\n[simula]: https://en.wikipedia.org/wiki/Simula\n\n\n\n## Superclasses and Subclasses\n\nGiven that the concept is \"inheritance\", you would hope they would pick a\nconsistent metaphor and call them \"parent\" and \"child\" classes, but that would\nbe too easy. Way back when, C. A. R. Hoare coined the term \"subclass\" to refer to a record type that refines another\ntype. Simula borrowed that term to refer to a *class* that inherits from\nanother. I don't think it was until Smalltalk came along that someone flipped\nthe Latin prefix to get \"superclass\" to refer to the other side of the\nrelationship. From C++, you also hear \"base\" and \"derived\" classes. I'll mostly\nstick with \"superclass\" and \"subclass\".\n\n\n\nOur first step towards supporting inheritance in Lox is a way to specify a\nsuperclass when declaring a class. There's a lot of variety in syntax for this.\nC++ and C# place a `:` after the subclass's name, followed by the superclass\nname. Java uses `extends` instead of the colon. Python puts the superclass(es)\nin parentheses after the class name. Simula puts the superclass's name *before*\nthe `class` keyword.\n\nThis late in the game, I'd rather not add a new reserved word or token to the\nlexer. We don't have `extends` or even `:`, so we'll follow Ruby and use a\nless-than sign (`<`).\n\n```lox\nclass Doughnut {\n // General doughnut stuff...\n}\n\nclass BostonCream < Doughnut {\n // Boston Cream-specific stuff...\n}\n```\n\nTo work this into the grammar, we add a new optional clause in our existing\n`classDecl` rule.\n\n```ebnf\nclassDecl → \"class\" IDENTIFIER ( \"<\" IDENTIFIER )?\n \"{\" function* \"}\" ;\n```\n\nAfter the class name, you can have a `<` followed by the superclass's name. The\nsuperclass clause is optional because you don't *have* to have a superclass.\nUnlike some other object-oriented languages like Java, Lox has no root \"Object\"\nclass that everything inherits from, so when you omit the superclass clause, the\nclass has *no* superclass, not even an implicit one.\n\nWe want to capture this new syntax in the class declaration's AST node.\n\n^code superclass-ast (1 before, 1 after)\n\nYou might be surprised that we store the superclass name as an Expr.Variable,\nnot a Token. The grammar restricts the superclass clause to a single identifier,\nbut at runtime, that identifier is evaluated as a variable access. Wrapping the\nname in an Expr.Variable early on in the parser gives us an object that the\nresolver can hang the resolution information off of.\n\nThe new parser code follows the grammar directly.\n\n^code parse-superclass (1 before, 1 after)\n\nOnce we've (possibly) parsed a superclass declaration, we store it in the AST.\n\n^code construct-class-ast (2 before, 1 after)\n\nIf we didn't parse a superclass clause, the superclass expression will be\n`null`. We'll have to make sure the later passes check for that. The first of\nthose is the resolver.\n\n^code resolve-superclass (1 before, 2 after)\n\nThe class declaration AST node has a new subexpression, so we traverse into and\nresolve that. Since classes are usually declared at the top level, the\nsuperclass name will most likely be a global variable, so this doesn't usually\ndo anything useful. However, Lox allows class declarations even inside blocks,\nso it's possible the superclass name refers to a local variable. In that case,\nwe need to make sure it's resolved.\n\nBecause even well-intentioned programmers sometimes write weird code, there's a\nsilly edge case we need to worry about while we're in here. Take a look at this:\n\n```lox\nclass Oops < Oops {}\n```\n\nThere's no way this will do anything useful, and if we let the runtime try to\nrun this, it will break the expectation the interpreter has about there not\nbeing cycles in the inheritance chain. The safest thing is to detect this case\nstatically and report it as an error.\n\n^code inherit-self (2 before, 1 after)\n\nAssuming the code resolves without error, the AST travels to the interpreter.\n\n^code interpret-superclass (1 before, 1 after)\n\nIf the class has a superclass expression, we evaluate it. Since that could\npotentially evaluate to some other kind of object, we have to check at runtime\nthat the thing we want to be the superclass is actually a class. Bad things\nwould happen if we allowed code like:\n\n```lox\nvar NotAClass = \"I am totally not a class\";\n\nclass Subclass < NotAClass {} // ?!\n```\n\nAssuming that check passes, we continue on. Executing a class declaration turns\nthe syntactic representation of a class -- its AST node -- into its runtime\nrepresentation, a LoxClass object. We need to plumb the superclass through to\nthat too. We pass the superclass to the constructor.\n\n^code interpreter-construct-class (3 before, 1 after)\n\nThe constructor stores it in a field.\n\n^code lox-class-constructor (1 after)\n\nWhich we declare here:\n\n^code lox-class-superclass-field (1 before, 1 after)\n\nWith that, we can define classes that are subclasses of other classes. Now, what\ndoes having a superclass actually *do?*\n\n## Inheriting Methods\n\nInheriting from another class means that everything that's true of the superclass should be true, more or less, of the\nsubclass. In statically typed languages, that carries a lot of implications. The\nsub*class* must also be a sub*type*, and the memory layout is controlled so that\nyou can pass an instance of a subclass to a function expecting a superclass and\nit can still access the inherited fields correctly.\n\n\n\nLox is a dynamically typed language, so our requirements are much simpler.\nBasically, it means that if you can call some method on an instance of the\nsuperclass, you should be able to call that method when given an instance of the\nsubclass. In other words, methods are inherited from the superclass.\n\nThis lines up with one of the goals of inheritance -- to give users a way to\nreuse code across classes. Implementing this in our interpreter is\nastonishingly easy.\n\n^code find-method-recurse-superclass (3 before, 1 after)\n\nThat's literally all there is to it. When we are looking up a method on an\ninstance, if we don't find it on the instance's class, we recurse up through the\nsuperclass chain and look there. Give it a try:\n\n```lox\nclass Doughnut {\n cook() {\n print \"Fry until golden brown.\";\n }\n}\n\nclass BostonCream < Doughnut {}\n\nBostonCream().cook();\n```\n\nThere we go, half of our inheritance features are complete with only three lines\nof Java code.\n\n## Calling Superclass Methods\n\nIn `findMethod()` we look for a method on the current class *before* walking up\nthe superclass chain. If a method with the same name exists in both the subclass\nand the superclass, the subclass one takes precedence or **overrides** the\nsuperclass method. Sort of like how variables in inner scopes shadow outer ones.\n\nThat's great if the subclass wants to *replace* some superclass behavior\ncompletely. But, in practice, subclasses often want to *refine* the superclass's\nbehavior. They want to do a little work specific to the subclass, but also\nexecute the original superclass behavior too.\n\nHowever, since the subclass has overridden the method, there's no way to refer\nto the original one. If the subclass method tries to call it by name, it will\njust recursively hit its own override. We need a way to say \"Call this method,\nbut look for it directly on my superclass and ignore my override\". Java uses\n`super` for this, and we'll use that same syntax in Lox. Here is an example:\n\n```lox\nclass Doughnut {\n cook() {\n print \"Fry until golden brown.\";\n }\n}\n\nclass BostonCream < Doughnut {\n cook() {\n super.cook();\n print \"Pipe full of custard and coat with chocolate.\";\n }\n}\n\nBostonCream().cook();\n```\n\nIf you run this, it should print:\n\n```text\nFry until golden brown.\nPipe full of custard and coat with chocolate.\n```\n\nWe have a new expression form. The `super` keyword, followed by a dot and an\nidentifier, looks for a method with that name. Unlike calls on `this`, the search\nstarts at the superclass.\n\n### Syntax\n\nWith `this`, the keyword works sort of like a magic variable, and the expression\nis that one lone token. But with `super`, the subsequent `.` and property name\nare inseparable parts of the `super` expression. You can't have a bare `super`\ntoken all by itself.\n\n```lox\nprint super; // Syntax error.\n```\n\nSo the new clause we add to the `primary` rule in our grammar includes the\nproperty access as well.\n\n```ebnf\nprimary → \"true\" | \"false\" | \"nil\" | \"this\"\n | NUMBER | STRING | IDENTIFIER | \"(\" expression \")\"\n | \"super\" \".\" IDENTIFIER ;\n```\n\nTypically, a `super` expression is used for a method call, but, as with regular\nmethods, the argument list is *not* part of the expression. Instead, a super\n*call* is a super *access* followed by a function call. Like other method calls,\nyou can get a handle to a superclass method and invoke it separately.\n\n```lox\nvar method = super.cook;\nmethod();\n```\n\nSo the `super` expression itself contains only the token for the `super` keyword\nand the name of the method being looked up. The corresponding syntax tree node is thus:\n\n^code super-expr (1 before, 1 after)\n\n\n\nFollowing the grammar, the new parsing code goes inside our existing `primary()`\nmethod.\n\n^code parse-super (2 before, 2 after)\n\nA leading `super` keyword tells us we've hit a `super` expression. After that we\nconsume the expected `.` and method name.\n\n### Semantics\n\nEarlier, I said a `super` expression starts the method lookup from \"the\nsuperclass\", but *which* superclass? The naïve answer is the superclass of\n`this`, the object the surrounding method was called on. That coincidentally\nproduces the right behavior in a lot of cases, but that's not actually correct.\nGaze upon:\n\n```lox\nclass A {\n method() {\n print \"A method\";\n }\n}\n\nclass B < A {\n method() {\n print \"B method\";\n }\n\n test() {\n super.method();\n }\n}\n\nclass C < B {}\n\nC().test();\n```\nTranslate this program to Java, C#, or C++ and it will print \"A method\", which\nis what we want Lox to do too. When this program runs, inside the body of\n`test()`, `this` is an instance of C. The superclass of C is B, but that is\n*not* where the lookup should start. If it did, we would hit B's `method()`.\n\nInstead, lookup should start on the superclass of *the class containing the\n`super` expression*. In this case, since `test()` is defined inside B, the\n`super` expression inside it should start the lookup on *B*’s superclass\n-- A.\n\n\n\n\"The\n\n\n\nThus, in order to evaluate a `super` expression, we need access to the\nsuperclass of the class definition surrounding the call. Alack and alas, at the\npoint in the interpreter where we are executing a `super` expression, we don't\nhave that easily available.\n\nWe *could* add a field to LoxFunction to store a reference to the LoxClass that\nowns that method. The interpreter would keep a reference to the\ncurrently executing LoxFunction so that we could look it up later when we hit a\n`super` expression. From there, we'd get the LoxClass of the method, then its\nsuperclass.\n\nThat's a lot of plumbing. In the [last chapter][], we had a similar problem when\nwe needed to add support for `this`. In that case, we used our existing\nenvironment and closure mechanism to store a reference to the current object.\nCould we do something similar for storing the superclass? Well, I probably wouldn't be talking about it if the\nanswer was no, so... yes.\n\n\n\n[last chapter]: classes.html\n\nOne important difference is that we bound `this` when the method was *accessed*.\nThe same method can be called on different instances and each needs its own\n`this`. With `super` expressions, the superclass is a fixed property of the\n*class declaration itself*. Every time you evaluate some `super` expression, the\nsuperclass is always the same.\n\nThat means we can create the environment for the superclass once, when the class\ndefinition is executed. Immediately before we define the methods, we make a new\nenvironment to bind the class's superclass to the name `super`.\n\n\"The\n\nWhen we create the LoxFunction runtime representation for each method, that is\nthe environment they will capture in their closure. Later, when a method is\ninvoked and `this` is bound, the superclass environment becomes the parent for\nthe method's environment, like so:\n\n\"The\n\nThat's a lot of machinery, but we'll get through it a step at a time. Before we\ncan get to creating the environment at runtime, we need to handle the\ncorresponding scope chain in the resolver.\n\n^code begin-super-scope (2 before, 2 after)\n\nIf the class declaration has a superclass, then we create a new scope\nsurrounding all of its methods. In that scope, we define the name \"super\". Once\nwe're done resolving the class's methods, we discard that scope.\n\n^code end-super-scope (2 before, 1 after)\n\nIt's a minor optimization, but we only create the superclass environment if the\nclass actually *has* a superclass. There's no point creating it when there isn't\na superclass since there'd be no superclass to store in it anyway.\n\nWith \"super\" defined in a scope chain, we are able to resolve the `super`\nexpression itself.\n\n^code resolve-super-expr\n\nWe resolve the `super` token exactly as if it were a variable. The resolution\nstores the number of hops along the environment chain that the interpreter needs\nto walk to find the environment where the superclass is stored.\n\nThis code is mirrored in the interpreter. When we evaluate a subclass\ndefinition, we create a new environment.\n\n^code begin-superclass-environment (6 before, 2 after)\n\nInside that environment, we store a reference to the superclass -- the actual\nLoxClass object for the superclass which we have now that we are in the runtime.\nThen we create the LoxFunctions for each method. Those will capture the current\nenvironment -- the one where we just bound \"super\" -- as their closure, holding\non to the superclass like we need. Once that's done, we pop the environment.\n\n^code end-superclass-environment (2 before, 2 after)\n\nWe're ready to interpret `super` expressions themselves. There are a few moving\nparts, so we'll build this method up in pieces.\n\n^code interpreter-visit-super\n\nFirst, the work we've been leading up to. We look up the surrounding class's\nsuperclass by looking up \"super\" in the proper environment.\n\nWhen we access a method, we also need to bind `this` to the object the method is\naccessed from. In an expression like `doughnut.cook`, the object is whatever we\nget from evaluating `doughnut`. In a `super` expression like `super.cook`, the\ncurrent object is implicitly the *same* current object that we're using. In\nother words, `this`. Even though we are looking up the *method* on the\nsuperclass, the *instance* is still `this`.\n\nUnfortunately, inside the `super` expression, we don't have a convenient node\nfor the resolver to hang the number of hops to `this` on. Fortunately, we do\ncontrol the layout of the environment chains. The environment where \"this\" is\nbound is always right inside the environment where we store \"super\".\n\n^code super-find-this (2 before, 1 after)\n\nOffsetting the distance by one looks up \"this\" in that inner environment. I\nadmit this isn't the most elegant code, but it\nworks.\n\n\n\nNow we're ready to look up and bind the method, starting at the superclass.\n\n^code super-find-method (2 before, 1 after)\n\nThis is almost exactly like the code for looking up a method of a get\nexpression, except that we call `findMethod()` on the superclass instead of on\nthe class of the current object.\n\nThat's basically it. Except, of course, that we might *fail* to find the method.\nSo we check for that too.\n\n^code super-no-method (2 before, 2 after)\n\nThere you have it! Take that BostonCream example earlier and give it a try.\nAssuming you and I did everything right, it should fry it first, then stuff it\nwith cream.\n\n### Invalid uses of super\n\nAs with previous language features, our implementation does the right thing when\nthe user writes correct code, but we haven't bulletproofed the intepreter\nagainst bad code. In particular, consider:\n\n```lox\nclass Eclair {\n cook() {\n super.cook();\n print \"Pipe full of crème pâtissière.\";\n }\n}\n```\n\nThis class has a `super` expression, but no superclass. At runtime, the code for\nevaluating `super` expressions assumes that \"super\" was successfully resolved\nand will be found in the environment. That's going to fail here because there is\nno surrounding environment for the superclass since there is no superclass. The\nJVM will throw an exception and bring our interpreter to its knees.\n\nHeck, there are even simpler broken uses of super:\n\n```lox\nsuper.notEvenInAClass();\n```\n\nWe could handle errors like these at runtime by checking to see if the lookup\nof \"super\" succeeded. But we can tell statically -- just by looking at the\nsource code -- that Eclair has no superclass and thus no `super` expression will\nwork inside it. Likewise, in the second example, we know that the `super`\nexpression is not even inside a method body.\n\nEven though Lox is dynamically typed, that doesn't mean we want to defer\n*everything* to runtime. If the user made a mistake, we'd like to help them find\nit sooner rather than later. So we'll report these errors statically, in the\nresolver.\n\nFirst, we add a new case to the enum we use to keep track of what kind of class\nis surrounding the current code being visited.\n\n^code class-type-subclass (1 before, 1 after)\n\nWe'll use that to distinguish when we're inside a class that has a superclass\nversus one that doesn't. When we resolve a class declaration, we set that if the\nclass is a subclass.\n\n^code set-current-subclass (1 before, 1 after)\n\nThen, when we resolve a `super` expression, we check to see that we are\ncurrently inside a scope where that's allowed.\n\n^code invalid-super (1 before, 1 after)\n\nIf not -- oopsie! -- the user made a mistake.\n\n## Conclusion\n\nWe made it! That final bit of error handling is the last chunk of code needed to\ncomplete our Java implementation of Lox. This is a real accomplishment and one you should be proud of. In the\npast dozen chapters and a thousand or so lines of code, we have learned and\nimplemented...\n\n* [tokens and lexing][4],\n* [abstract syntax trees][5],\n* [recursive descent parsing][6],\n* prefix and infix expressions,\n* runtime representation of objects,\n* [interpreting code using the Visitor pattern][7],\n* [lexical scope][8],\n* environment chains for storing variables,\n* [control flow][9],\n* [functions with parameters][10],\n* closures,\n* [static variable resolution and error detection][11],\n* [classes][12],\n* constructors,\n* fields,\n* methods, and finally,\n* inheritance.\n\n[4]: scanning.html\n[5]: representing-code.html\n[6]: parsing-expressions.html\n[7]: evaluating-expressions.html\n[8]: statements-and-state.html\n[9]: control-flow.html\n[10]: functions.html\n[11]: resolving-and-binding.html\n[12]: classes.html\n\n\n\nWe did all of that from scratch, with no external dependencies or magic tools.\nJust you and I, our respective text editors, a couple of collection classes in\nthe Java standard library, and the JVM runtime.\n\nThis marks the end of Part II, but not the end of the book. Take a break. Maybe\nwrite a few fun Lox programs and run them in your interpreter. (You may want to\nadd a few more native methods for things like reading user input.) When you're\nrefreshed and ready, we'll embark on our [next adventure][].\n\n[next adventure]: a-bytecode-virtual-machine.html\n\n
\n\n## Challenges\n\n1. Lox supports only *single inheritance* -- a class may have a single\n superclass and that's the only way to reuse methods across classes. Other\n languages have explored a variety of ways to more freely reuse and share\n capabilities across classes: mixins, traits, multiple inheritance, virtual\n inheritance, extension methods, etc.\n\n If you were to add some feature along these lines to Lox, which would you\n pick and why? If you're feeling courageous (and you should be at this\n point), go ahead and add it.\n\n1. In Lox, as in most other object-oriented languages, when looking up a\n method, we start at the bottom of the class hierarchy and work our way up --\n a subclass's method is preferred over a superclass's. In order to get to the\n superclass method from within an overriding method, you use `super`.\n\n The language [BETA][] takes the [opposite approach][inner]. When you call a\n method, it starts at the *top* of the class hierarchy and works *down*. A\n superclass method wins over a subclass method. In order to get to the\n subclass method, the superclass method can call `inner`, which is sort of\n like the inverse of `super`. It chains to the next method down the\n hierarchy.\n\n The superclass method controls when and where the subclass is allowed to\n refine its behavior. If the superclass method doesn't call `inner` at all,\n then the subclass has no way of overriding or modifying the superclass's\n behavior.\n\n Take out Lox's current overriding and `super` behavior and replace it with\n BETA's semantics. In short:\n\n * When calling a method on a class, prefer the method *highest* on the\n class's inheritance chain.\n\n * Inside the body of a method, a call to `inner` looks for a method with\n the same name in the nearest subclass along the inheritance chain\n between the class containing the `inner` and the class of `this`. If\n there is no matching method, the `inner` call does nothing.\n\n For example:\n\n ```lox\n class Doughnut {\n cook() {\n print \"Fry until golden brown.\";\n inner();\n print \"Place in a nice box.\";\n }\n }\n\n class BostonCream < Doughnut {\n cook() {\n print \"Pipe full of custard and coat with chocolate.\";\n }\n }\n\n BostonCream().cook();\n ```\n\n This should print:\n\n ```text\n Fry until golden brown.\n Pipe full of custard and coat with chocolate.\n Place in a nice box.\n ```\n\n1. In the chapter where I introduced Lox, [I challenged you][challenge] to\n come up with a couple of features you think the language is missing. Now\n that you know how to build an interpreter, implement one of those features.\n\n[challenge]: the-lox-language.html#challenges\n[inner]: http://journal.stuffwithstuff.com/2012/12/19/the-impoliteness-of-overriding-methods/\n[beta]: https://beta.cs.au.dk/\n\n
\n" }, { "path": "book/introduction.md", "content": "> Fairy tales are more than true: not because they tell us that dragons exist,\n> but because they tell us that dragons can be beaten.\n>\n> G.K. Chesterton by way of Neil Gaiman, Coraline\n\nI'm really excited we're going on this journey together. This is a book on\nimplementing interpreters for programming languages. It's also a book on how to\ndesign a language worth implementing. It's the book I wish I'd had when I first\nstarted getting into languages, and it's the book I've been writing in my head for nearly a decade.\n\n\n\nIn these pages, we will walk step-by-step through two complete interpreters for\na full-featured language. I assume this is your first foray into languages, so\nI'll cover each concept and line of code you need to build a complete, usable,\nfast language implementation.\n\nIn order to cram two full implementations inside one book without it turning\ninto a doorstop, this text is lighter on theory than others. As we build each\npiece of the system, I will introduce the history and concepts behind it. I'll\ntry to get you familiar with the lingo so that if you ever find yourself at a\ncocktail party full of PL (programming language)\nresearchers, you'll fit in.\n\n\n\nBut we're mostly going to spend our brain juice getting the language up and\nrunning. This is not to say theory isn't important. Being able to reason\nprecisely and formally about syntax and semantics is\na vital skill when working on a language. But, personally, I learn best by\ndoing. It's hard for me to wade through paragraphs full of abstract concepts and\nreally absorb them. But if I've coded something, run it, and debugged it, then I\n*get* it.\n\n\n\nThat's my goal for you. I want you to come away with a solid intuition of how a\nreal language lives and breathes. My hope is that when you read other, more\ntheoretical books later, the concepts there will firmly stick in your mind,\nadhered to this tangible substrate.\n\n## Why Learn This Stuff?\n\nEvery introduction to every compiler book seems to have this section. I don't\nknow what it is about programming languages that causes such existential doubt.\nI don't think ornithology books worry about justifying their existence. They\nassume the reader loves birds and start teaching.\n\nBut programming languages are a little different. I suppose it is true that the\nodds of any of us creating a broadly successful, general-purpose programming\nlanguage are slim. The designers of the world's widely used languages could fit\nin a Volkswagen bus, even without putting the pop-top camper up. If joining that\nelite group was the *only* reason to learn languages, it would be hard to\njustify. Fortunately, it isn't.\n\n### Little languages are everywhere\n\nFor every successful general-purpose language, there are a thousand successful\nniche ones. We used to call them \"little languages\", but inflation in the jargon\neconomy led to the name \"domain-specific languages\". These are pidgins\ntailor-built to a specific task. Think application scripting languages, template\nengines, markup formats, and configuration files.\n\n\"A\n\n\n\nAlmost every large software project needs a handful of these. When you can, it's\ngood to reuse an existing one instead of rolling your own. Once you factor in\ndocumentation, debuggers, editor support, syntax highlighting, and all of the\nother trappings, doing it yourself becomes a tall order.\n\nBut there's still a good chance you'll find yourself needing to whip up a parser\nor other tool when there isn't an existing library that fits your needs. Even\nwhen you are reusing some existing implementation, you'll inevitably end up\nneeding to debug and maintain it and poke around in its guts.\n\n### Languages are great exercise\n\nLong distance runners sometimes train with weights strapped to their ankles or\nat high altitudes where the atmosphere is thin. When they later unburden\nthemselves, the new relative ease of light limbs and oxygen-rich air enables\nthem to run farther and faster.\n\nImplementing a language is a real test of programming skill. The code is complex\nand performance critical. You must master recursion, dynamic arrays, trees,\ngraphs, and hash tables. You probably use hash tables at least in your\nday-to-day programming, but do you *really* understand them? Well, after we've\ncrafted our own from scratch, I guarantee you will.\n\nWhile I intend to show you that an interpreter isn't as daunting as you might\nbelieve, implementing one well is still a challenge. Rise to it, and you'll come\naway a stronger programmer, and smarter about how you use data structures and\nalgorithms in your day job.\n\n### One more reason\n\nThis last reason is hard for me to admit, because it's so close to my heart.\nEver since I learned to program as a kid, I felt there was something magical\nabout languages. When I first tapped out BASIC programs one key at a time I\ncouldn't conceive how BASIC *itself* was made.\n\nLater, the mixture of awe and terror on my college friends' faces when talking\nabout their compilers class was enough to convince me language hackers were a\ndifferent breed of human -- some sort of wizards granted privileged access to\narcane arts.\n\nIt's a charming image, but it has a darker side. *I*\ndidn't feel like a wizard, so I was left thinking I lacked some inborn quality\nnecessary to join the cabal. Though I've been fascinated by languages ever since\nI doodled made-up keywords in my school notebook, it took me decades to muster\nthe courage to try to really learn them. That \"magical\" quality, that sense of\nexclusivity, excluded *me*.\n\n\n\nWhen I did finally start cobbling together my own little interpreters, I quickly\nlearned that, of course, there is no magic at all. It's just code, and the\npeople who hack on languages are just people.\n\nThere *are* a few techniques you don't often encounter outside of languages, and\nsome parts are a little difficult. But not more difficult than other obstacles\nyou've overcome. My hope is that if you've felt intimidated by languages and\nthis book helps you overcome that fear, maybe I'll leave you just a tiny bit\nbraver than you were before.\n\nAnd, who knows, maybe you *will* make the next great language. Someone has to.\n\n## How the Book Is Organized\n\nThis book is broken into three parts. You're reading the first one now. It's a\ncouple of chapters to get you oriented, teach you some of the lingo that\nlanguage hackers use, and introduce you to Lox, the language we'll be\nimplementing.\n\nEach of the other two parts builds one complete Lox interpreter. Within those\nparts, each chapter is structured the same way. The chapter takes a single\nlanguage feature, teaches you the concepts behind it, and walks you through an\nimplementation.\n\nIt took a good bit of trial and error on my part, but I managed to carve up the\ntwo interpreters into chapter-sized chunks that build on the previous chapters\nbut require nothing from later ones. From the very first chapter, you'll have a\nworking program you can run and play with. With each passing chapter, it grows\nincreasingly full-featured until you eventually have a complete language.\n\nAside from copious, scintillating English prose, chapters have a few other\ndelightful facets:\n\n### The code\n\nWe're about *crafting* interpreters, so this book contains real code. Every\nsingle line of code needed is included, and each snippet tells you where to\ninsert it in your ever-growing implementation.\n\nMany other language books and language implementations use tools like [Lex][]\nand [Yacc][], so-called **compiler-compilers**, that\nautomatically generate some of the source files for an implementation from some\nhigher-level description. There are pros and cons to tools like those, and\nstrong opinions -- some might say religious convictions -- on both sides.\n\n\n\nWe will abstain from using them here. I want to ensure there are no dark corners\nwhere magic and confusion can hide, so we'll write everything by hand. As you'll\nsee, it's not as bad as it sounds, and it means you really will understand each\nline of code and how both interpreters work.\n\n[lex]: https://en.wikipedia.org/wiki/Lex_(software)\n[yacc]: https://en.wikipedia.org/wiki/Yacc\n\nA book has different constraints from the \"real world\" and so the coding style\nhere might not always reflect the best way to write maintainable production\nsoftware. If I seem a little cavalier about, say, omitting `private` or\ndeclaring a global variable, understand I do so to keep the code easier on your\neyes. The pages here aren't as wide as your IDE and every character counts.\n\nAlso, the code doesn't have many comments. That's because each handful of lines\nis surrounded by several paragraphs of honest-to-God prose explaining it. When\nyou write a book to accompany your program, you are welcome to omit comments\ntoo. Otherwise, you should probably use `//` a little more than I do.\n\nWhile the book contains every line of code and teaches what each means, it does\nnot describe the machinery needed to compile and run the interpreter. I assume\nyou can slap together a makefile or a project in your IDE of choice in order to\nget the code to run. Those kinds of instructions get out of date quickly, and\nI want this book to age like XO brandy, not backyard hooch.\n\n### Snippets\n\nSince the book contains literally every line of code needed for the\nimplementations, the snippets are quite precise. Also, because I try to keep the\nprogram in a runnable state even when major features are missing, sometimes we\nadd temporary code that gets replaced in later snippets.\n\nA snippet with all the bells and whistles looks like this:\n\n
\n      default:\n
lox/Scanner.java
\nin scanToken()
\nreplace 1 line
\n
\n        if (isDigit(c)) {\n          number();\n        } else {\n          Lox.error(line, "Unexpected character.");\n        }\n
\n        break;\n
\n
lox/Scanner.java, in scanToken(), replace 1 line
\n\nIn the center, you have the new code to add. It may have a few faded out lines\nabove or below to show where it goes in the existing surrounding code. There is\nalso a little blurb telling you in which file and where to place the snippet. If\nthat blurb says \"replace _ lines\", there is some existing code between the faded\nlines that you need to remove and replace with the new snippet.\n\n### Asides\n\nAsides contain biographical sketches, historical\nbackground, references to related topics, and suggestions of other areas to\nexplore. There's nothing that you *need* to know in them to understand later\nparts of the book, so you can skip them if you want. I won't judge you, but I\nmight be a little sad.\n\n\n\n### Challenges\n\nEach chapter ends with a few exercises. Unlike textbook problem sets, which tend\nto review material you already covered, these are to help you learn *more* than\nwhat's in the chapter. They force you to step off the guided path and explore on\nyour own. They will make you research other languages, figure out how to\nimplement features, or otherwise get you out of your comfort zone.\n\nVanquish the challenges and you'll come away with a\nbroader understanding and possibly a few bumps and scrapes. Or skip them if you\nwant to stay inside the comfy confines of the tour bus. It's your book.\n\n\n\n### Design notes\n\nMost \"programming language\" books are strictly programming language\n*implementation* books. They rarely discuss how one might happen to *design* the\nlanguage being implemented. Implementation is fun because it is so precisely defined. We programmers seem to have an\naffinity for things that are black and white, ones and zeroes.\n\n\n\nPersonally, I think the world needs only so many implementations of FORTRAN 77. At some point, you find yourself designing a\n*new* language. Once you start playing *that* game, then the softer, human side\nof the equation becomes paramount. Things like which features are easy to learn,\nhow to balance innovation and familiarity, what syntax is more readable and to\nwhom.\n\n\n\nAll of that stuff profoundly affects the success of your new language. I want\nyour language to succeed, so in some chapters I end with a \"design note\", a\nlittle essay on some corner of the human aspect of programming languages. I'm no\nexpert on this -- I don't know if anyone really is -- so take these with a large\npinch of salt. That should make them tastier food for thought, which is my main\naim.\n\n## The First Interpreter\n\nWe'll write our first interpreter, jlox, in Java. The\nfocus is on *concepts*. We'll write the simplest, cleanest code we can to\ncorrectly implement the semantics of the language. This will get us comfortable\nwith the basic techniques and also hone our understanding of exactly how the\nlanguage is supposed to behave.\n\n\n\nJava is a great language for this. It's high level enough that we don't get\noverwhelmed by fiddly implementation details, but it's still pretty explicit.\nUnlike in scripting languages, there tends to be less complex machinery hiding\nunder the hood, and you've got static types to see what data structures you're\nworking with.\n\nI also chose Java specifically because it is an object-oriented language. That\nparadigm swept the programming world in the '90s and is now the dominant way of\nthinking for millions of programmers. Odds are good you're already used to\norganizing code into classes and methods, so we'll keep you in that comfort\nzone.\n\nWhile academic language folks sometimes look down on object-oriented languages,\nthe reality is that they are widely used even for language work. GCC and LLVM\nare written in C++, as are most JavaScript virtual machines. Object-oriented\nlanguages are ubiquitous, and the tools and compilers *for* a language are often\nwritten *in* the same language.\n\n\n\nAnd, finally, Java is hugely popular. That means there's a good chance you\nalready know it, so there's less for you to learn to get going in the book. If\nyou aren't that familiar with Java, don't freak out. I try to stick to a fairly\nminimal subset of it. I use the diamond operator from Java 7 to make things a\nlittle more terse, but that's about it as far as \"advanced\" features go. If you\nknow another object-oriented language, like C# or C++, you can muddle through.\n\nBy the end of part II, we'll have a simple, readable implementation. It's not\nvery fast, but it's correct. However, we are only able to accomplish that by\nbuilding on the Java virtual machine's own runtime facilities. We want to learn\nhow Java *itself* implements those things.\n\n## The Second Interpreter\n\nSo in the next part, we start all over again, but this time in C. C is the\nperfect language for understanding how an implementation *really* works, all the\nway down to the bytes in memory and the code flowing through the CPU.\n\nA big reason that we're using C is so I can show you things C is particularly\ngood at, but that *does* mean you'll need to be pretty comfortable with it. You\ndon't have to be the reincarnation of Dennis Ritchie, but you shouldn't be\nspooked by pointers either.\n\nIf you aren't there yet, pick up an introductory book on C and chew through it,\nthen come back here when you're done. In return, you'll come away from this book\nan even stronger C programmer. That's useful given how many language\nimplementations are written in C: Lua, CPython, and Ruby's MRI, to name a few.\n\nIn our C interpreter, clox, we are forced to implement\nfor ourselves all the things Java gave us for free. We'll write our own dynamic\narray and hash table. We'll decide how objects are represented in memory, and\nbuild a garbage collector to reclaim them.\n\n\n\nOur Java implementation was focused on being correct. Now that we have that\ndown, we'll turn to also being *fast*. Our C interpreter will contain a compiler that translates Lox to an efficient bytecode\nrepresentation (don't worry, I'll get into what that means soon), which it then\nexecutes. This is the same technique used by implementations of Lua, Python,\nRuby, PHP, and many other successful languages.\n\n\n\nWe'll even try our hand at benchmarking and optimization. By the end, we'll have\na robust, accurate, fast interpreter for our language, able to keep up with\nother professional caliber implementations out there. Not bad for one book and a\nfew thousand lines of code.\n\n
\n\n## Challenges\n\n1. There are at least six domain-specific languages used in the [little system\n I cobbled together][repo] to write and publish this book. What are they?\n\n1. Get a \"Hello, world!\" program written and running in Java. Set up whatever\n makefiles or IDE projects you need to get it working. If you have a\n debugger, get comfortable with it and step through your program as it runs.\n\n1. Do the same thing for C. To get some practice with pointers, define a\n [doubly linked list][] of heap-allocated strings. Write functions to insert,\n find, and delete items from it. Test them.\n\n[repo]: https://github.com/munificent/craftinginterpreters\n[doubly linked list]: https://en.wikipedia.org/wiki/Doubly_linked_list\n\n
\n\n
\n\n## Design Note: What's in a Name?\n\nOne of the hardest challenges in writing this book was coming up with a name for\nthe language it implements. I went through *pages* of candidates before I found\none that worked. As you'll discover on the first day you start building your own\nlanguage, naming is deviously hard. A good name satisfies a few criteria:\n\n1. **It isn't in use.** You can run into all sorts of trouble, legal and\n social, if you inadvertently step on someone else's name.\n\n2. **It's easy to pronounce.** If things go well, hordes of people will be\n saying and writing your language's name. Anything longer than a couple of\n syllables or a handful of letters will annoy them to no end.\n\n3. **It's distinct enough to search for.** People will Google your language's\n name to learn about it, so you want a word that's rare enough that most\n results point to your docs. Though, with the amount of AI search engines are\n packing today, that's less of an issue. Still, you won't be doing your users\n any favors if you name your language \"for\".\n\n4. **It doesn't have negative connotations across a number of cultures.** This\n is hard to be on guard for, but it's worth considering. The designer of\n Nimrod ended up renaming his language to \"Nim\" because too many people\n remember that Bugs Bunny used \"Nimrod\" as an insult. (Bugs was using it\n ironically.)\n\nIf your potential name makes it through that gauntlet, keep it. Don't get hung\nup on trying to find an appellation that captures the quintessence of your\nlanguage. If the names of the world's other successful languages teach us\nanything, it's that the name doesn't matter much. All you need is a reasonably\nunique token.\n\n
\n" }, { "path": "book/jumping-back-and-forth.md", "content": "> The order that our mind imagines is like a net, or like a ladder, built to\n> attain something. But afterward you must throw the ladder away, because you\n> discover that, even if it was useful, it was meaningless.\n>\n> Umberto Eco, The Name of the Rose\n\nIt's taken a while to get here, but we're finally ready to add control flow to\nour virtual machine. In the tree-walk interpreter we built for jlox, we\nimplemented Lox's control flow in terms of Java's. To execute a Lox `if`\nstatement, we used a Java `if` statement to run the chosen branch. That works,\nbut isn't entirely satisfying. By what magic does the *JVM itself* or a native\nCPU implement `if` statements? Now that we have our own bytecode VM to hack on,\nwe can answer that.\n\nWhen we talk about \"control flow\", what are we referring to? By \"flow\" we mean\nthe way execution moves through the text of the program. Almost like there is a\nlittle robot inside the computer wandering through our code, executing bits and\npieces here and there. Flow is the path that robot takes, and by *controlling*\nthe robot, we drive which pieces of code it executes.\n\nIn jlox, the robot's locus of attention -- the *current* bit of code -- was\nimplicit based on which AST nodes were stored in various Java variables and what\nJava code we were in the middle of running. In clox, it is much more explicit.\nThe VM's `ip` field stores the address of the current bytecode instruction. The\nvalue of that field is exactly \"where we are\" in the program.\n\nExecution proceeds normally by incrementing the `ip`. But we can mutate that\nvariable however we want to. In order to implement control flow, all that's\nnecessary is to change the `ip` in more interesting ways. The simplest control\nflow construct is an `if` statement with no `else` clause:\n\n```lox\nif (condition) print(\"condition was truthy\");\n```\n\nThe VM evaluates the bytecode for the condition expression. If the result is\ntruthy, then it continues along and executes the `print` statement in the body.\nThe interesting case is when the condition is falsey. When that happens,\nexecution skips over the then branch and proceeds to the next statement.\n\nTo skip over a chunk of code, we simply set the `ip` field to the address of the\nbytecode instruction following that code. To *conditionally* skip over some\ncode, we need an instruction that looks at the value on top of the stack. If\nit's falsey, it adds a given offset to the `ip` to jump over a range of\ninstructions. Otherwise, it does nothing and lets execution proceed to the next\ninstruction as usual.\n\nWhen we compile to bytecode, the explicit nested block structure of the code\nevaporates, leaving only a flat series of instructions behind. Lox is a\n[structured programming][] language, but clox bytecode isn't. The right -- or\nwrong, depending on how you look at it -- set of bytecode instructions could\njump into the middle of a block, or from one scope into another.\n\nThe VM will happily execute that, even if the result leaves the stack in an\nunknown, inconsistent state. So even though the bytecode is unstructured, we'll\ntake care to ensure that our compiler only generates clean code that maintains\nthe same structure and nesting that Lox itself does.\n\nThis is exactly how real CPUs behave. Even though we might program them using\nhigher-level languages that mandate structured control flow, the compiler lowers\nthat down to raw jumps. At the bottom, it turns out goto is the only real\ncontrol flow.\n\n[structured programming]: https://en.wikipedia.org/wiki/Structured_programming\n\nAnyway, I didn't mean to get all philosophical. The important bit is that if we\nhave that one conditional jump instruction, that's enough to implement Lox's\n`if` statement, as long as it doesn't have an `else` clause. So let's go ahead\nand get started with that.\n\n## If Statements\n\nThis many chapters in, you know the drill. Any new feature starts in the front\nend and works its way through the pipeline. An `if` statement is, well, a\nstatement, so that's where we hook it into the parser.\n\n^code parse-if (2 before, 1 after)\n\nWhen we see an `if` keyword, we hand off compilation to this function:\n\n^code if-statement\n\n\n\nFirst we compile the condition expression, bracketed by parentheses. At runtime,\nthat will leave the condition value on top of the stack. We'll use that to\ndetermine whether to execute the then branch or skip it.\n\nThen we emit a new `OP_JUMP_IF_FALSE` instruction. It has an operand for how\nmuch to offset the `ip` -- how many bytes of code to skip. If the condition is\nfalsey, it adjusts the `ip` by that amount. Something like this:\n\n\n\n\n\n\"Flowchart\n\nBut we have a problem. When we're writing the `OP_JUMP_IF_FALSE` instruction's\noperand, how do we know how far to jump? We haven't compiled the then branch\nyet, so we don't know how much bytecode it contains.\n\nTo fix that, we use a classic trick called **backpatching**. We emit the jump\ninstruction first with a placeholder offset operand. We keep track of where that\nhalf-finished instruction is. Next, we compile the then body. Once that's done,\nwe know how far to jump. So we go back and replace that placeholder offset with\nthe real one now that we can calculate it. Sort of like sewing a patch onto the\nexisting fabric of the compiled code.\n\n\"A\n\nWe encode this trick into two helper functions.\n\n^code emit-jump\n\nThe first emits a bytecode instruction and writes a placeholder operand for the\njump offset. We pass in the opcode as an argument because later we'll have two\ndifferent instructions that use this helper. We use two bytes for the jump\noffset operand. A 16-bit offset lets us jump over up\nto 65,535 bytes of code, which should be plenty for our needs.\n\n\n\nThe function returns the offset of the emitted instruction in the chunk. After\ncompiling the then branch, we take that offset and pass it to this:\n\n^code patch-jump\n\nThis goes back into the bytecode and replaces the operand at the given location\nwith the calculated jump offset. We call `patchJump()` right before we emit the\nnext instruction that we want the jump to land on, so it uses the current\nbytecode count to determine how far to jump. In the case of an `if` statement,\nthat means right after we compile the then branch and before we compile the next\nstatement.\n\nThat's all we need at compile time. Let's define the new instruction.\n\n^code jump-if-false-op (1 before, 1 after)\n\nOver in the VM, we get it working like so:\n\n^code op-jump-if-false (2 before, 1 after)\n\nThis is the first instruction we've added that takes a 16-bit operand. To read\nthat from the chunk, we use a new macro.\n\n^code read-short (1 before, 1 after)\n\nIt yanks the next two bytes from the chunk and builds a 16-bit unsigned integer\nout of them. As usual, we clean up our macro when we're done with it.\n\n^code undef-read-short (1 before, 1 after)\n\nAfter reading the offset, we check the condition value on top of the stack.\nIf it's falsey, we apply this jump offset to the `ip`.\nOtherwise, we leave the `ip` alone and execution will automatically proceed to\nthe next instruction following the jump instruction.\n\nIn the case where the condition is falsey, we don't need to do any other work.\nWe've offset the `ip`, so when the outer instruction dispatch loop turns again,\nit will pick up execution at that new instruction, past all of the code in the\nthen branch.\n\n\n\nNote that the jump instruction doesn't pop the condition value off the stack. So\nwe aren't totally done here, since this leaves an extra value floating around on\nthe stack. We'll clean that up soon. Ignoring that for the moment, we do have a\nworking `if` statement in Lox now, with only one little instruction required to\nsupport it at runtime in the VM.\n\n### Else clauses\n\nAn `if` statement without support for `else` clauses is like Morticia Addams\nwithout Gomez. So, after we compile the then branch, we look for an `else`\nkeyword. If we find one, we compile the else branch.\n\n^code compile-else (1 before, 1 after)\n\nWhen the condition is falsey, we'll jump over the then branch. If there's an\nelse branch, the `ip` will land right at the beginning of its code. But that's\nnot enough, though. Here's the flow that leads to:\n\n\"Flowchart\n\nIf the condition is truthy, we execute the then branch like we want. But after\nthat, execution rolls right on through into the else branch. Oops! When the\ncondition is true, after we run the then branch, we need to jump over the else\nbranch. That way, in either case, we only execute a single branch, like this:\n\n\"Flowchart\n\nTo implement that, we need another jump from the end of the then branch.\n\n^code jump-over-else (2 before, 1 after)\n\nWe patch that offset after the end of the else body.\n\n^code patch-else (1 before, 1 after)\n\nAfter executing the then branch, this jumps to the next statement after the else\nbranch. Unlike the other jump, this jump is unconditional. We always take it, so\nwe need another instruction that expresses that.\n\n^code jump-op (1 before, 1 after)\n\nWe interpret it like so:\n\n^code op-jump (2 before, 1 after)\n\nNothing too surprising here -- the only difference is that it doesn't check a\ncondition and always applies the offset.\n\nWe have then and else branches working now, so we're close. The last bit is to\nclean up that condition value we left on the stack. Remember, each statement is\nrequired to have zero stack effect -- after the statement is finished executing,\nthe stack should be as tall as it was before.\n\nWe could have the `OP_JUMP_IF_FALSE` instruction pop the condition itself, but\nsoon we'll use that same instruction for the logical operators where we don't\nwant the condition popped. Instead, we'll have the compiler emit a couple of\nexplicit `OP_POP` instructions when compiling an `if` statement. We need to take\ncare that every execution path through the generated code pops the condition.\n\nWhen the condition is truthy, we pop it right before the code inside the then\nbranch.\n\n^code pop-then (1 before, 1 after)\n\nOtherwise, we pop it at the beginning of the else branch.\n\n^code pop-end (1 before, 2 after)\n\nThis little instruction here also means that every `if` statement has an\nimplicit else branch even if the user didn't write an `else` clause. In the case\nwhere they left it off, all the branch does is discard the condition value.\n\nThe full correct flow looks like this:\n\n\"Flowchart\n\nIf you trace through, you can see that it always executes a single branch and\nensures the condition is popped first. All that remains is a little disassembler\nsupport.\n\n^code disassemble-jump (1 before, 1 after)\n\nThese two instructions have a new format with a 16-bit operand, so we add a new\nutility function to disassemble them.\n\n^code jump-instruction\n\nThere we go, that's one complete control flow construct. If this were an '80s\nmovie, the montage music would kick in and the rest of the control flow syntax\nwould take care of itself. Alas, the '80s are long over,\nso we'll have to grind it out ourselves.\n\n\n\n## Logical Operators\n\nYou probably remember this from jlox, but the logical operators `and` and `or`\naren't just another pair of binary operators like `+` and `-`. Because they\nshort-circuit and may not evaluate their right operand depending on the value of\nthe left one, they work more like control flow expressions.\n\nThey're basically a little variation on an `if` statement with an `else` clause.\nThe easiest way to explain them is to just show you the compiler code and the\ncontrol flow it produces in the resulting bytecode. Starting with `and`, we hook\nit into the expression parsing table here:\n\n^code table-and (1 before, 1 after)\n\nThat hands off to a new parser function.\n\n^code and\n\nAt the point this is called, the left-hand side expression has already been\ncompiled. That means at runtime, its value will be on top of the stack. If that\nvalue is falsey, then we know the entire `and` must be false, so we skip the\nright operand and leave the left-hand side value as the result of the entire\nexpression. Otherwise, we discard the left-hand value and evaluate the right\noperand which becomes the result of the whole `and` expression.\n\nThose four lines of code right there produce exactly that. The flow looks like\nthis:\n\n\"Flowchart\n\nNow you can see why `OP_JUMP_IF_FALSE` leaves the\nvalue on top of the stack. When the left-hand side of the `and` is falsey, that\nvalue sticks around to become the result of the entire expression.\n\n\n\n### Logical or operator\n\nThe `or` operator is a little more complex. First we add it to the parse table.\n\n^code table-or (1 before, 1 after)\n\nWhen that parser consumes an infix `or` token, it calls this:\n\n^code or\n\nIn an `or` expression, if the left-hand side is *truthy*, then we skip over the\nright operand. Thus we need to jump when a value is truthy. We could add a\nseparate instruction, but just to show how our compiler is free to map the\nlanguage's semantics to whatever instruction sequence it wants, I implemented it\nin terms of the jump instructions we already have.\n\nWhen the left-hand side is falsey, it does a tiny jump over the next statement.\nThat statement is an unconditional jump over the code for the right operand.\nThis little dance effectively does a jump when the value is truthy. The flow\nlooks like this:\n\n\"Flowchart\n\nIf I'm honest with you, this isn't the best way to do this. There are more\ninstructions to dispatch and more overhead. There's no good reason why `or`\nshould be slower than `and`. But it is kind of fun to see that it's possible to\nimplement both operators without adding any new instructions. Forgive me my\nindulgences.\n\nOK, those are the three *branching* constructs in Lox. By that, I mean, these\nare the control flow features that only jump *forward* over code. Other\nlanguages often have some kind of multi-way branching statement like `switch`\nand maybe a conditional expression like `?:`, but Lox keeps it simple.\n\n## While Statements\n\nThat takes us to the *looping* statements, which jump *backward* so that code\ncan be executed more than once. Lox only has two loop constructs, `while` and\n`for`. A `while` loop is (much) simpler, so we start the party there.\n\n^code parse-while (1 before, 1 after)\n\nWhen we reach a `while` token, we call:\n\n^code while-statement\n\nMost of this mirrors `if` statements -- we compile the condition expression,\nsurrounded by mandatory parentheses. That's followed by a jump instruction that\nskips over the subsequent body statement if the condition is falsey.\n\nWe patch the jump after compiling the body and take care to pop the condition value from the stack on either path. The\nonly difference from an `if` statement is the loop. That looks like this:\n\n\n\n^code loop (1 before, 2 after)\n\nAfter the body, we call this function to emit a \"loop\" instruction. That\ninstruction needs to know how far back to jump. When jumping forward, we had to\nemit the instruction in two stages since we didn't know how far we were going to\njump until after we emitted the jump instruction. We don't have that problem\nnow. We've already compiled the point in code that we want to jump back to --\nit's right before the condition expression.\n\nAll we need to do is capture that location as we compile it.\n\n^code loop-start (1 before, 1 after)\n\nAfter executing the body of a `while` loop, we jump all the way back to before\nthe condition. That way, we re-evaluate the condition expression on each\niteration. We store the chunk's current instruction count in `loopStart` to\nrecord the offset in the bytecode right before the condition expression we're\nabout to compile. Then we pass that into this helper function:\n\n^code emit-loop\n\nIt's a bit like `emitJump()` and `patchJump()` combined. It emits a new loop\ninstruction, which unconditionally jumps *backwards* by a given offset. Like the\njump instructions, after that we have a 16-bit operand. We calculate the offset\nfrom the instruction we're currently at to the `loopStart` point that we want to\njump back to. The `+ 2` is to take into account the size of the `OP_LOOP`\ninstruction's own operands which we also need to jump over.\n\nFrom the VM's perspective, there really is no semantic difference between\n`OP_LOOP` and `OP_JUMP`. Both just add an offset to the `ip`. We could have used\na single instruction for both and given it a signed offset operand. But I\nfigured it was a little easier to sidestep the annoying bit twiddling required\nto manually pack a signed 16-bit integer into two bytes, and we've got the\nopcode space available, so why not use it?\n\nThe new instruction is here:\n\n^code loop-op (1 before, 1 after)\n\nAnd in the VM, we implement it thusly:\n\n^code op-loop (1 before, 1 after)\n\nThe only difference from `OP_JUMP` is a subtraction instead of an addition.\nDisassembly is similar too.\n\n^code disassemble-loop (1 before, 1 after)\n\nThat's our `while` statement. It contains two jumps -- a conditional forward one\nto escape the loop when the condition is not met, and an unconditional loop\nbackward after we have executed the body. The flow looks like this:\n\n\"Flowchart\n\n## For Statements\n\nThe other looping statement in Lox is the venerable `for` loop, inherited from\nC. It's got a lot more going on with it compared to a `while` loop. It has three\nclauses, all of which are optional:\n\n\n\n* The initializer can be a variable declaration or an expression. It runs once\n at the beginning of the statement.\n\n* The condition clause is an expression. Like in a `while` loop, we exit the\n loop when it evaluates to something falsey.\n\n* The increment expression runs once at the end of each loop iteration.\n\n\n\nIn jlox, the parser desugared a `for` loop to a synthesized AST for a `while`\nloop with some extra stuff before it and at the end of the body. We'll do\nsomething similar, though we won't go through anything like an AST. Instead,\nour bytecode compiler will use the jump and loop instructions we already have.\n\nWe'll work our way through the implementation a piece at a time, starting with\nthe `for` keyword.\n\n^code parse-for (1 before, 1 after)\n\nIt calls a helper function. If we only supported `for` loops with empty clauses\nlike `for (;;)`, then we could implement it like this:\n\n^code for-statement\n\nThere's a bunch of mandatory punctuation at the top. Then we compile the body.\nLike we did for `while` loops, we record the bytecode offset at the top of the\nbody and emit a loop to jump back to that point after it. We've got a working\nimplementation of infinite loops now.\n\n\n\n### Initializer clause\n\nNow we'll add the first clause, the initializer. It executes only once, before\nthe body, so compiling is straightforward.\n\n^code for-initializer (1 before, 2 after)\n\nThe syntax is a little complex since we allow either a variable declaration or\nan expression. We use the presence of the `var` keyword to tell which we have.\nFor the expression case, we call `expressionStatement()` instead of\n`expression()`. That looks for a semicolon, which we need here too, and also\nemits an `OP_POP` instruction to discard the value. We don't want the\ninitializer to leave anything on the stack.\n\nIf a `for` statement declares a variable, that variable should be scoped to the\nloop body. We ensure that by wrapping the whole statement in a scope.\n\n^code for-begin-scope (1 before, 1 after)\n\nThen we close it at the end.\n\n^code for-end-scope (1 before, 1 after)\n\n### Condition clause\n\nNext, is the condition expression that can be used to exit the loop.\n\n^code for-exit (1 before, 1 after)\n\nSince the clause is optional, we need to see if it's actually present. If the\nclause is omitted, the next token must be a semicolon, so we look for that to\ntell. If there isn't a semicolon, there must be a condition expression.\n\nIn that case, we compile it. Then, just like with while, we emit a conditional\njump that exits the loop if the condition is falsey. Since the jump leaves the\nvalue on the stack, we pop it before executing the body. That ensures we discard\nthe value when the condition is true.\n\nAfter the loop body, we need to patch that jump.\n\n^code exit-jump (1 before, 2 after)\n\nWe do this only when there is a condition clause. If there isn't, there's no\njump to patch and no condition value on the stack to pop.\n\n### Increment clause\n\nI've saved the best for last, the increment clause. It's pretty convoluted. It\nappears textually before the body, but executes *after* it. If we parsed to an\nAST and generated code in a separate pass, we could simply traverse into and\ncompile the `for` statement AST's body field before its increment clause.\n\nUnfortunately, we can't compile the increment clause later, since our compiler\nonly makes a single pass over the code. Instead, we'll *jump over* the\nincrement, run the body, jump *back* up to the increment, run it, and then go to\nthe next iteration.\n\nI know, a little weird, but hey, it beats manually managing ASTs in memory in C,\nright? Here's the code:\n\n^code for-increment (2 before, 2 after)\n\nAgain, it's optional. Since this is the last clause, when omitted, the next\ntoken will be the closing parenthesis. When an increment is present, we need to\ncompile it now, but it shouldn't execute yet. So, first, we emit an\nunconditional jump that hops over the increment clause's code to the body of the\nloop.\n\nNext, we compile the increment expression itself. This is usually an assignment.\nWhatever it is, we only execute it for its side effect, so we also emit a pop to\ndiscard its value.\n\nThe last part is a little tricky. First, we emit a loop instruction. This is the\nmain loop that takes us back to the top of the `for` loop -- right before the\ncondition expression if there is one. That loop happens right after the\nincrement, since the increment executes at the end of each loop iteration.\n\nThen we change `loopStart` to point to the offset where the increment expression\nbegins. Later, when we emit the loop instruction after the body statement, this\nwill cause it to jump up to the *increment* expression instead of the top of the\nloop like it does when there is no increment. This is how we weave the\nincrement in to run after the body.\n\nIt's convoluted, but it all works out. A complete loop with all the clauses\ncompiles to a flow like this:\n\n\"Flowchart\n\nAs with implementing `for` loops in jlox, we didn't need to touch the runtime.\nIt all gets compiled down to primitive control flow operations the VM already\nsupports. In this chapter, we've taken a big leap\nforward -- clox is now Turing complete. We've also covered quite a bit of new\nsyntax: three statements and two expression forms. Even so, it only took three\nnew simple instructions. That's a pretty good effort-to-reward ratio for the\narchitecture of our VM.\n\n\n\n
\n\n## Challenges\n\n1. In addition to `if` statements, most C-family languages have a multi-way\n `switch` statement. Add one to clox. The grammar is:\n\n ```ebnf\n switchStmt → \"switch\" \"(\" expression \")\"\n \"{\" switchCase* defaultCase? \"}\" ;\n switchCase → \"case\" expression \":\" statement* ;\n defaultCase → \"default\" \":\" statement* ;\n ```\n\n To execute a `switch` statement, first evaluate the parenthesized switch\n value expression. Then walk the cases. For each case, evaluate its value\n expression. If the case value is equal to the switch value, execute the\n statements under the case and then exit the `switch` statement. Otherwise,\n try the next case. If no case matches and there is a `default` clause,\n execute its statements.\n\n To keep things simpler, we're omitting fallthrough and `break` statements.\n Each case automatically jumps to the end of the switch statement after its\n statements are done.\n\n1. In jlox, we had a challenge to add support for `break` statements. This\n time, let's do `continue`:\n\n ```ebnf\n continueStmt → \"continue\" \";\" ;\n ```\n\n A `continue` statement jumps directly to the top of the nearest enclosing\n loop, skipping the rest of the loop body. Inside a `for` loop, a `continue`\n jumps to the increment clause, if there is one. It's a compile-time error to\n have a `continue` statement not enclosed in a loop.\n\n Make sure to think about scope. What should happen to local variables\n declared inside the body of the loop or in blocks nested inside the loop\n when a `continue` is executed?\n\n1. Control flow constructs have been mostly unchanged since Algol 68. Language\n evolution since then has focused on making code more declarative and high\n level, so imperative control flow hasn't gotten much attention.\n\n For fun, try to invent a useful novel control flow feature for Lox. It can\n be a refinement of an existing form or something entirely new. In practice,\n it's hard to come up with something useful enough at this low expressiveness\n level to outweigh the cost of forcing a user to learn an unfamiliar notation\n and behavior, but it's a good chance to practice your design skills.\n\n
\n\n
\n\n## Design Note: Considering Goto Harmful\n\nDiscovering that all of our beautiful structured control flow in Lox is actually\ncompiled to raw unstructured jumps is like the moment in Scooby Doo when the\nmonster rips the mask off their face. It was goto all along! Except in this\ncase, the monster is *under* the mask. We all know goto is evil. But... why?\n\nIt is true that you can write outrageously unmaintainable code using goto. But I\ndon't think most programmers around today have seen that first hand. It's been a\nlong time since that style was common. These days, it's a boogie man we invoke\nin scary stories around the campfire.\n\nThe reason we rarely confront that monster in person is because Edsger Dijkstra\nslayed it with his famous letter \"Go To Statement Considered Harmful\", published\nin *Communications of the ACM* (March, 1968). Debate around structured\nprogramming had been fierce for some time with adherents on both sides, but I\nthink Dijkstra deserves the most credit for effectively ending it. Most new\nlanguages today have no unstructured jump statements.\n\nA one-and-a-half page letter that almost single-handedly destroyed a language\nfeature must be pretty impressive stuff. If you haven't read it, I encourage you\nto do so. It's a seminal piece of computer science lore, one of our tribe's\nancestral songs. Also, it's a nice, short bit of practice for reading academic\nCS writing, which is a useful skill to develop.\n\n\n\nI've read it through a number of times, along with a few critiques, responses,\nand commentaries. I ended up with mixed feelings, at best. At a very high level,\nI'm with him. His general argument is something like this:\n\n1. As programmers, we write programs -- static text -- but what we care about\n is the actual running program -- its dynamic behavior.\n\n2. We're better at reasoning about static things than dynamic things. (He\n doesn't provide any evidence to support this claim, but I accept it.)\n\n3. Thus, the more we can make the dynamic execution of the program reflect its\n textual structure, the better.\n\nThis is a good start. Drawing our attention to the separation between the code\nwe write and the code as it runs inside the machine is an interesting insight.\nThen he tries to define a \"correspondence\" between program text and execution.\nFor someone who spent literally his entire career advocating greater rigor in\nprogramming, his definition is pretty hand-wavey. He says:\n\n> Let us now consider how we can characterize the progress of a process. (You\n> may think about this question in a very concrete manner: suppose that a\n> process, considered as a time succession of actions, is stopped after an\n> arbitrary action, what data do we have to fix in order that we can redo the\n> process until the very same point?)\n\nImagine it like this. You have two computers with the same program running on\nthe exact same inputs -- so totally deterministic. You pause one of them at an\narbitrary point in its execution. What data would you need to send to the other\ncomputer to be able to stop it exactly as far along as the first one was?\n\nIf your program allows only simple statements like assignment, it's easy. You\njust need to know the point after the last statement you executed. Basically a\nbreakpoint, the `ip` in our VM, or the line number in an error message. Adding\nbranching control flow like `if` and `switch` doesn't add any more to this. Even\nif the marker points inside a branch, we can still tell where we are.\n\nOnce you add function calls, you need something more. You could have paused the\nfirst computer in the middle of a function, but that function may be called from\nmultiple places. To pause the second machine at exactly the same point in *the\nentire program's* execution, you need to pause it on the *right* call to that\nfunction.\n\nSo you need to know not just the current statement, but, for function calls that\nhaven't returned yet, you need to know the locations of the callsites. In other\nwords, a call stack, though I don't think that term existed when Dijkstra wrote\nthis. Groovy.\n\nHe notes that loops make things harder. If you pause in the middle of a loop\nbody, you don't know how many iterations have run. So he says you also need to\nkeep an iteration count. And, since loops can nest, you need a stack of those\n(presumably interleaved with the call stack pointers since you can be in loops\nin outer calls too).\n\nThis is where it gets weird. So we're really building to something now, and you\nexpect him to explain how goto breaks all of this. Instead, he just says:\n\n> The unbridled use of the go to statement has an immediate consequence that it\n> becomes terribly hard to find a meaningful set of coordinates in which to\n> describe the process progress.\n\nHe doesn't prove that this is hard, or say why. He just says it. He does say\nthat one approach is unsatisfactory:\n\n> With the go to statement one can, of course, still describe the progress\n> uniquely by a counter counting the number of actions performed since program\n> start (viz. a kind of normalized clock). The difficulty is that such a\n> coordinate, although unique, is utterly unhelpful.\n\nBut... that's effectively what loop counters do, and he was fine with those.\nIt's not like every loop is a simple \"for every integer from 0 to 10\"\nincrementing count. Many are `while` loops with complex conditionals.\n\nTaking an example close to home, consider the core bytecode execution loop at\nthe heart of clox. Dijkstra argues that that loop is tractable because we can\nsimply count how many times the loop has run to reason about its progress. But\nthat loop runs once for each executed instruction in some user's compiled Lox\nprogram. Does knowing that it executed 6,201 bytecode instructions really tell\nus VM maintainers *anything* edifying about the state of the interpreter?\n\nIn fact, this particular example points to a deeper truth. Böhm and Jacopini\n[proved][] that *any* control flow using goto can be transformed into one using\njust sequencing, loops, and branches. Our bytecode interpreter loop is a living\nexample of that proof: it implements the unstructured control flow of the clox\nbytecode instruction set without using any gotos itself.\n\n[proved]: https://en.wikipedia.org/wiki/Structured_program_theorem\n\nThat seems to offer a counter-argument to Dijkstra's claim: you *can* define a\ncorrespondence for a program using gotos by transforming it to one that doesn't\nand then use the correspondence from that program, which -- according to him --\nis acceptable because it uses only branches and loops.\n\nBut, honestly, my argument here is also weak. I think both of us are basically\ndoing pretend math and using fake logic to make what should be an empirical,\nhuman-centered argument. Dijkstra is right that some code using goto is really\nbad. Much of that could and should be turned into clearer code by using\nstructured control flow.\n\nBy eliminating goto completely from languages, you're definitely prevented from\nwriting bad code using gotos. It may be that forcing users to use structured\ncontrol flow and making it an uphill battle to write goto-like code using those\nconstructs is a net win for all of our productivity.\n\nBut I do wonder sometimes if we threw out the baby with the bathwater. In the\nabsence of goto, we often resort to more complex structured patterns. The\n\"switch inside a loop\" is a classic one. Another is using a guard variable to\nexit out of a series of nested loops:\n\n\n\n\n```c\n// See if the matrix contains a zero.\nbool found = false;\nfor (int x = 0; x < xSize; x++) {\n for (int y = 0; y < ySize; y++) {\n for (int z = 0; z < zSize; z++) {\n if (matrix[x][y][z] == 0) {\n printf(\"found\");\n found = true;\n break;\n }\n }\n if (found) break;\n }\n if (found) break;\n}\n```\n\nIs that really better than:\n\n```c\nfor (int x = 0; x < xSize; x++) {\n for (int y = 0; y < ySize; y++) {\n for (int z = 0; z < zSize; z++) {\n if (matrix[x][y][z] == 0) {\n printf(\"found\");\n goto done;\n }\n }\n }\n}\ndone:\n```\n\n\n\nI guess what I really don't like is that we're making language design and\nengineering decisions today based on fear. Few people today have any subtle\nunderstanding of the problems and benefits of goto. Instead, we just think it's\n\"considered harmful\". Personally, I've never found dogma a good starting place\nfor quality creative work.\n\n
\n" }, { "path": "book/local-variables.md", "content": "> And as imagination bodies forth
\n> The forms of things unknown, the poet's pen
\n> Turns them to shapes and gives to airy nothing
\n> A local habitation and a name.\n>\n> William Shakespeare, A Midsummer Night's Dream\n\nThe [last chapter][] introduced variables to clox, but only of the global variety. In this chapter, we'll extend that to\nsupport blocks, block scope, and local variables. In jlox, we managed to pack\nall of that and globals into one chapter. For clox, that's two chapters worth of\nwork partially because, frankly, everything takes more effort in C.\n\n\n\n[last chapter]: global-variables.html\n\nBut an even more important reason is that our approach to local variables will\nbe quite different from how we implemented globals. Global variables are late\nbound in Lox. \"Late\" in this context means \"resolved after compile time\". That's\ngood for keeping the compiler simple, but not great for performance. Local\nvariables are one of the most-used parts of a\nlanguage. If locals are slow, *everything* is slow. So we want a strategy for\nlocal variables that's as efficient as possible.\n\n\n\nFortunately, lexical scoping is here to help us. As the name implies, lexical\nscope means we can resolve a local variable just by looking at the text of the\nprogram -- locals are *not* late bound. Any processing work we do in the\ncompiler is work we *don't* have to do at runtime, so our implementation of\nlocal variables will lean heavily on the compiler.\n\n## Representing Local Variables\n\nThe nice thing about hacking on a programming language in modern times is\nthere's a long lineage of other languages to learn from. So how do C and Java\nmanage their local variables? Why, on the stack, of course! They typically use\nthe native stack mechanisms supported by the chip and OS. That's a little too\nlow level for us, but inside the virtual world of clox, we have our own stack we\ncan use.\n\nRight now, we only use it for holding on to **temporaries** -- short-lived blobs\nof data that we need to remember while computing an expression. As long as we\ndon't get in the way of those, we can stuff our local variables onto the stack\ntoo. This is great for performance. Allocating space for a new local requires\nonly incrementing the `stackTop` pointer, and freeing is likewise a decrement.\nAccessing a variable from a known stack slot is an indexed array lookup.\n\nWe do need to be careful, though. The VM expects the stack to behave like, well,\na stack. We have to be OK with allocating new locals only on the top of the\nstack, and we have to accept that we can discard a local only when nothing is\nabove it on the stack. Also, we need to make sure temporaries don't interfere.\n\nConveniently, the design of Lox is in harmony with\nthese constraints. New locals are always created by declaration statements.\nStatements don't nest inside expressions, so there are never any temporaries on\nthe stack when a statement begins executing. Blocks are strictly nested. When a\nblock ends, it always takes the innermost, most recently declared locals with\nit. Since those are also the locals that came into scope last, they should be on\ntop of the stack where we need them.\n\n\n\nStep through this example program and watch how the local variables come in and\ngo out of scope:\n\n\"A\n\nSee how they fit a stack perfectly? It seems that the stack will work for\nstoring locals at runtime. But we can go further than that. Not only do we know\n*that* they will be on the stack, but we can even pin down precisely *where*\nthey will be on the stack. Since the compiler knows exactly which local\nvariables are in scope at any point in time, it can effectively simulate the\nstack during compilation and note where in the stack each\nvariable lives.\n\nWe'll take advantage of this by using these stack offsets as operands for the\nbytecode instructions that read and store local variables. This makes working\nwith locals deliciously fast -- as simple as indexing into an array.\n\n\n\nThere's a lot of state we need to track in the compiler to make this whole thing\ngo, so let's get started there. In jlox, we used a linked chain of \"environment\"\nHashMaps to track which local variables were currently in scope. That's sort of\nthe classic, schoolbook way of representing lexical scope. For clox, as usual,\nwe're going a little closer to the metal. All of the state lives in a new\nstruct.\n\n^code compiler-struct (1 before, 2 after)\n\nWe have a simple, flat array of all locals that are in scope during each point in\nthe compilation process. They are ordered in the array\nin the order that their declarations appear in the code. Since the instruction\noperand we'll use to encode a local is a single byte, our VM has a hard limit on\nthe number of locals that can be in scope at once. That means we can also give\nthe locals array a fixed size.\n\n\n\n^code uint8-count (1 before, 2 after)\n\nBack in the Compiler struct, the `localCount` field tracks how many locals are\nin scope -- how many of those array slots are in use. We also track the \"scope\ndepth\". This is the number of blocks surrounding the current bit of code we're\ncompiling.\n\nOur Java interpreter used a chain of maps to keep each block's variables\nseparate from other blocks'. This time, we'll simply number variables with the\nlevel of nesting where they appear. Zero is the global scope, one is the first\ntop-level block, two is inside that, you get the idea. We use this to track\nwhich block each local belongs to so that we know which locals to discard when a\nblock ends.\n\nEach local in the array is one of these:\n\n^code local-struct (1 before, 2 after)\n\nWe store the name of the variable. When we're resolving an identifier, we\ncompare the identifier's lexeme with each local's name to find a match. It's\npretty hard to resolve a variable if you don't know its name. The `depth` field\nrecords the scope depth of the block where the local variable was declared.\nThat's all the state we need for now.\n\nThis is a very different representation from what we had in jlox, but it still\nlets us answer all of the same questions our compiler needs to ask of the\nlexical environment. The next step is figuring out how the compiler *gets* at\nthis state. If we were principled engineers, we'd\ngive each function in the front end a parameter that accepts a pointer to a\nCompiler. We'd create a Compiler at the beginning and carefully thread it\nthrough each function call... but that would mean a lot of boring changes to\nthe code we already wrote, so here's a global variable instead:\n\n\n\n^code current-compiler (1 before, 1 after)\n\nHere's a little function to initialize the compiler:\n\n^code init-compiler\n\nWhen we first start up the VM, we call it to get everything into a clean state.\n\n^code compiler (1 before, 1 after)\n\nOur compiler has the data it needs, but not the operations on that data. There's\nno way to create and destroy scopes, or add and resolve variables. We'll add\nthose as we need them. First, let's start building some language features.\n\n## Block Statements\n\nBefore we can have any local variables, we need some local scopes. These come\nfrom two things: function bodies and blocks. Functions\nare a big chunk of work that we'll tackle in [a later chapter][functions], so\nfor now we're only going to do blocks. As usual, we start with the syntax. The\nnew grammar we'll introduce is:\n\n```ebnf\nstatement → exprStmt\n | printStmt\n | block ;\n\nblock → \"{\" declaration* \"}\" ;\n```\n\n\n\nBlocks are a kind of statement, so the rule for them goes in the `statement`\nproduction. The corresponding code to compile one looks like this:\n\n^code parse-block (2 before, 1 after)\n\nAfter parsing the initial curly brace, we use this\nhelper function to compile the rest of the block:\n\n\n\n^code block\n\nIt keeps parsing declarations and statements until it hits the closing brace. As\nwe do with any loop in the parser, we also check for the end of the token\nstream. This way, if there's a malformed program with a missing closing curly,\nthe compiler doesn't get stuck in a loop.\n\nExecuting a block simply means executing the statements it contains, one after\nthe other, so there isn't much to compiling them. The semantically interesting\nthing blocks do is create scopes. Before we compile the body of a block, we call\nthis function to enter a new local scope:\n\n^code begin-scope\n\nIn order to \"create\" a scope, all we do is increment the current depth. This is\ncertainly much faster than jlox, which allocated an entire new HashMap for\neach one. Given `beginScope()`, you can probably guess what `endScope()` does.\n\n^code end-scope\n\nThat's it for blocks and scopes -- more or less -- so we're ready to stuff some\nvariables into them.\n\n## Declaring Local Variables\n\nUsually we start with parsing here, but our compiler already supports parsing\nand compiling variable declarations. We've got `var` statements, identifier\nexpressions and assignment in there now. It's just that the compiler assumes\nall variables are global. So we don't need any new parsing support, we just need\nto hook up the new scoping semantics to the existing code.\n\n\"The\n\nVariable declaration parsing begins in `varDeclaration()` and relies on a couple\nof other functions. First, `parseVariable()` consumes the identifier token for\nthe variable name, adds its lexeme to the chunk's constant table as a string,\nand then returns the constant table index where it was added. Then, after\n`varDeclaration()` compiles the initializer, it calls `defineVariable()` to emit\nthe bytecode for storing the variable's value in the global variable hash table.\n\nBoth of those helpers need a few changes to support local variables. In\n`parseVariable()`, we add:\n\n^code parse-local (1 before, 1 after)\n\nFirst, we \"declare\" the variable. I'll get to what that means in a second. After\nthat, we exit the function if we're in a local scope. At runtime, locals aren't\nlooked up by name. There's no need to stuff the variable's name into the\nconstant table, so if the declaration is inside a local scope, we return a dummy\ntable index instead.\n\nOver in `defineVariable()`, we need to emit the code to store a local variable\nif we're in a local scope. It looks like this:\n\n^code define-variable (1 before, 1 after)\n\nWait, what? Yup. That's it. There is no code to create a local variable at\nruntime. Think about what state the VM is in. It has already executed the code\nfor the variable's initializer (or the implicit `nil` if the user omitted an\ninitializer), and that value is sitting right on top of the stack as the only\nremaining temporary. We also know that new locals are allocated at the top of\nthe stack... right where that value already is. Thus, there's nothing to do. The\ntemporary simply *becomes* the local variable. It doesn't get much more\nefficient than that.\n\n\n\n\"Walking\n\n\n\nOK, so what's \"declaring\" about? Here's what that does:\n\n^code declare-variable\n\nThis is the point where the compiler records the existence of the variable. We\nonly do this for locals, so if we're in the top-level global scope, we just bail\nout. Because global variables are late bound, the compiler doesn't keep track of\nwhich declarations for them it has seen.\n\nBut for local variables, the compiler does need to remember that the variable\nexists. That's what declaring it does -- it adds it to the compiler's list of\nvariables in the current scope. We implement that using another new function.\n\n^code add-local\n\nThis initializes the next available Local in the compiler's array of variables.\nIt stores the variable's name and the depth of the\nscope that owns the variable.\n\n\n\nOur implementation is fine for a correct Lox program, but what about invalid\ncode? Let's aim to be robust. The first error to handle is not really the user's\nfault, but more a limitation of the VM. The instructions to work with local\nvariables refer to them by slot index. That index is stored in a single-byte\noperand, which means the VM only supports up to 256 local variables in scope at\none time.\n\nIf we try to go over that, not only could we not refer to them at runtime, but\nthe compiler would overwrite its own locals array, too. Let's prevent that.\n\n^code too-many-locals (1 before, 1 after)\n\nThe next case is trickier. Consider:\n\n```lox\n{\n var a = \"first\";\n var a = \"second\";\n}\n```\n\nAt the top level, Lox allows redeclaring a variable with the same name as a\nprevious declaration because that's useful for the REPL. But inside a local\nscope, that's a pretty weird thing to do. It's likely\nto be a mistake, and many languages, including our own Lox, enshrine that\nassumption by making this an error.\n\n\n\nNote that the above program is different from this one:\n\n```lox\n{\n var a = \"outer\";\n {\n var a = \"inner\";\n }\n}\n```\n\nIt's OK to have two variables with the same name in *different* scopes, even\nwhen the scopes overlap such that both are visible at the same time. That's\nshadowing, and Lox does allow that. It's only an error to have two variables\nwith the same name in the *same* local scope.\n\nWe detect that error like so:\n\n^code existing-in-scope (1 before, 2 after)\n\n\n\nLocal variables are appended to the array when they're declared, which means the\ncurrent scope is always at the end of the array. When we declare a new variable,\nwe start at the end and work backward, looking for an existing variable with the\nsame name. If we find one in the current scope, we report the error. Otherwise,\nif we reach the beginning of the array or a variable owned by another scope,\nthen we know we've checked all of the existing variables in the scope.\n\nTo see if two identifiers are the same, we use this:\n\n^code identifiers-equal\n\nSince we know the lengths of both lexemes, we check that first. That will fail\nquickly for many non-equal strings. If the lengths are\nthe same, we check the characters using `memcmp()`. To get to `memcmp()`, we\nneed an include.\n\n\n\n^code compiler-include-string (1 before, 2 after)\n\nWith this, we're able to bring variables into being. But, like ghosts, they\nlinger on beyond the scope where they are declared. When a block ends, we need\nto put them to rest.\n\n^code pop-locals (1 before, 1 after)\n\nWhen we pop a scope, we walk backward through the local array looking for any\nvariables declared at the scope depth we just left. We discard them by simply\ndecrementing the length of the array.\n\nThere is a runtime component to this too. Local variables occupy slots on the\nstack. When a local variable goes out of scope, that slot is no longer needed\nand should be freed. So, for each variable that we discard, we also emit an\n`OP_POP` instruction to pop it from the stack.\n\n\n\n## Using Locals\n\nWe can now compile and execute local variable declarations. At runtime, their\nvalues are sitting where they should be on the stack. Let's start using them.\nWe'll do both variable access and assignment at the same time since they touch\nthe same functions in the compiler.\n\nWe already have code for getting and setting global variables, and -- like good\nlittle software engineers -- we want to reuse as much of that existing code as\nwe can. Something like this:\n\n^code named-local (1 before, 2 after)\n\nInstead of hardcoding the bytecode instructions emitted for variable access and\nassignment, we use a couple of C variables. First, we try to find a local\nvariable with the given name. If we find one, we use the instructions for\nworking with locals. Otherwise, we assume it's a global variable and use the\nexisting bytecode instructions for globals.\n\nA little further down, we use those variables to emit the right instructions.\nFor assignment:\n\n^code emit-set (2 before, 1 after)\n\nAnd for access:\n\n^code emit-get (2 before, 1 after)\n\nThe real heart of this chapter, the part where we resolve a local variable, is\nhere:\n\n^code resolve-local\n\nFor all that, it's straightforward. We walk the list of locals that are\ncurrently in scope. If one has the same name as the identifier token, the\nidentifier must refer to that variable. We've found it! We walk the array\nbackward so that we find the *last* declared variable with the identifier. That\nensures that inner local variables correctly shadow locals with the same name in\nsurrounding scopes.\n\nAt runtime, we load and store locals using the stack slot index, so that's what\nthe compiler needs to calculate after it resolves the variable. Whenever a\nvariable is declared, we append it to the locals array in Compiler. That means\nthe first local variable is at index zero, the next one is at index one, and so\non. In other words, the locals array in the compiler has the *exact* same layout\nas the VM's stack will have at runtime. The variable's index in the locals array\nis the same as its stack slot. How convenient!\n\nIf we make it through the whole array without finding a variable with the given\nname, it must not be a local. In that case, we return `-1` to signal that it\nwasn't found and should be assumed to be a global variable instead.\n\n### Interpreting local variables\n\nOur compiler is emitting two new instructions, so let's get them working. First\nis loading a local variable:\n\n^code get-local-op (1 before, 1 after)\n\nAnd its implementation:\n\n^code interpret-get-local (1 before, 1 after)\n\nIt takes a single-byte operand for the stack slot where the local lives. It\nloads the value from that index and then pushes it on top of the stack where\nlater instructions can find it.\n\n\n\nNext is assignment:\n\n^code set-local-op (1 before, 1 after)\n\nYou can probably predict the implementation.\n\n^code interpret-set-local (1 before, 1 after)\n\nIt takes the assigned value from the top of the stack and stores it in the stack\nslot corresponding to the local variable. Note that it doesn't pop the value\nfrom the stack. Remember, assignment is an expression, and every expression\nproduces a value. The value of an assignment expression is the assigned value\nitself, so the VM just leaves the value on the stack.\n\nOur disassembler is incomplete without support for these two new instructions.\n\n^code disassemble-local (1 before, 1 after)\n\nThe compiler compiles local variables to direct slot access. The local\nvariable's name never leaves the compiler to make it into the chunk at all.\nThat's great for performance, but not so great for introspection. When we\ndisassemble these instructions, we can't show the variable's name like we could\nwith globals. Instead, we just show the slot number.\n\n\n\n^code byte-instruction\n\n### Another scope edge case\n\nWe already sunk some time into handling a couple of weird edge cases around\nscopes. We made sure shadowing works correctly. We report an error if two\nvariables in the same local scope have the same name. For reasons that aren't\nentirely clear to me, variable scoping seems to have a lot of these wrinkles.\nI've never seen a language where it feels completely elegant.\n\n\n\nWe've got one more edge case to deal with before we end this chapter. Recall this strange beastie we first met in [jlox's implementation of variable resolution][shadow]:\n\n[shadow]: resolving-and-binding.html#resolving-variable-declarations\n\n```lox\n{\n var a = \"outer\";\n {\n var a = a;\n }\n}\n```\n\nWe slayed it then by splitting a variable's declaration into two phases, and\nwe'll do that again here:\n\n\"An\n\nAs soon as the variable declaration begins -- in other words, before its\ninitializer -- the name is declared in the current scope. The variable exists,\nbut in a special \"uninitialized\" state. Then we compile the initializer. If at\nany point in that expression we resolve an identifier that points back to this\nvariable, we'll see that it is not initialized yet and report an error. After we\nfinish compiling the initializer, we mark the variable as initialized and ready\nfor use.\n\nTo implement this, when we declare a local, we need to indicate the\n\"uninitialized\" state somehow. We could add a new field to Local, but let's be a\nlittle more parsimonious with memory. Instead, we'll set the variable's scope\ndepth to a special sentinel value, `-1`.\n\n^code declare-undefined (1 before, 1 after)\n\nLater, once the variable's initializer has been compiled, we mark it\ninitialized.\n\n^code define-local (1 before, 2 after)\n\nThat is implemented like so:\n\n^code mark-initialized\n\nSo this is *really* what \"declaring\" and \"defining\" a variable means in the\ncompiler. \"Declaring\" is when the variable is added to the scope, and \"defining\"\nis when it becomes available for use.\n\nWhen we resolve a reference to a local variable, we check the scope depth to see\nif it's fully defined.\n\n^code own-initializer-error (1 before, 1 after)\n\nIf the variable has the sentinel depth, it must be a reference to a variable in\nits own initializer, and we report that as an error.\n\nThat's it for this chapter! We added blocks, local variables, and real,\nhonest-to-God lexical scoping. Given that we introduced an entirely different\nruntime representation for variables, we didn't have to write a lot of code. The\nimplementation ended up being pretty clean and efficient.\n\nYou'll notice that almost all of the code we wrote is in the compiler. Over in\nthe runtime, it's just two little instructions. You'll see this as a continuing\ntrend in clox compared to jlox. One of the biggest\nhammers in the optimizer's toolbox is pulling work forward into the compiler so\nthat you don't have to do it at runtime. In this chapter, that meant resolving\nexactly which stack slot every local variable occupies. That way, at runtime, no\nlookup or resolution needs to happen.\n\n\n\n
\n\n## Challenges\n\n1. Our simple local array makes it easy to calculate the stack slot of each\n local variable. But it means that when the compiler resolves a reference to\n a variable, we have to do a linear scan through the array.\n\n Come up with something more efficient. Do you think the additional\n complexity is worth it?\n\n1. How do other languages handle code like this:\n\n ```lox\n var a = a;\n ```\n\n What would you do if it was your language? Why?\n\n1. Many languages make a distinction between variables that can be reassigned\n and those that can't. In Java, the `final` modifier prevents you from\n assigning to a variable. In JavaScript, a variable declared with `let` can\n be assigned, but one declared using `const` can't. Swift treats `let` as\n single-assignment and uses `var` for assignable variables. Scala and Kotlin\n use `val` and `var`.\n\n Pick a keyword for a single-assignment variable form to add to Lox. Justify\n your choice, then implement it. An attempt to assign to a variable declared\n using your new keyword should cause a compile error.\n\n1. Extend clox to allow more than 256 local variables to be in scope at a time.\n\n
\n" }, { "path": "book/methods-and-initializers.md", "content": "> When you are on the dancefloor, there is nothing to do but dance.\n>\n> Umberto Eco, The Mysterious Flame of Queen Loana\n\nIt is time for our virtual machine to bring its nascent objects to life with\nbehavior. That means methods and method calls. And, since they are a special\nkind of method, initializers too.\n\nAll of this is familiar territory from our previous jlox interpreter. What's new\nin this second trip is an important optimization we'll implement to make method\ncalls over seven times faster than our baseline performance. But before we get\nto that fun, we gotta get the basic stuff working.\n\n## Method Declarations\n\nWe can't optimize method calls before we have method calls, and we can't call\nmethods without having methods to call, so we'll start with declarations.\n\n### Representing methods\n\nWe usually start in the compiler, but let's knock the object model out first\nthis time. The runtime representation for methods in clox is similar to that of\njlox. Each class stores a hash table of methods. Keys are method names, and each\nvalue is an ObjClosure for the body of the method.\n\n^code class-methods (3 before, 1 after)\n\nA brand new class begins with an empty method table.\n\n^code init-methods (1 before, 1 after)\n\nThe ObjClass struct owns the memory for this table, so when the memory manager\ndeallocates a class, the table should be freed too.\n\n^code free-methods (1 before, 1 after)\n\nSpeaking of memory managers, the GC needs to trace through classes into the\nmethod table. If a class is still reachable (likely through some instance),\nthen all of its methods certainly need to stick around too.\n\n^code mark-methods (1 before, 1 after)\n\nWe use the existing `markTable()` function, which traces through the key string\nand value in each table entry.\n\nStoring a class's methods is pretty familiar coming from jlox. The different\npart is how that table gets populated. Our previous interpreter had access to\nthe entire AST node for the class declaration and all of the methods it\ncontained. At runtime, the interpreter simply walked that list of declarations.\n\nNow every piece of information the compiler wants to shunt over to the runtime\nhas to squeeze through the interface of a flat series of bytecode instructions.\nHow do we take a class declaration, which can contain an arbitrarily large set\nof methods, and represent it as bytecode? Let's hop over to the compiler and\nfind out.\n\n### Compiling method declarations\n\nThe last chapter left us with a compiler that parses classes but allows only an\nempty body. Now we insert a little code to compile a series of method\ndeclarations between the braces.\n\n^code class-body (1 before, 1 after)\n\nLox doesn't have field declarations, so anything before the closing brace at the\nend of the class body must be a method. We stop compiling methods when we hit\nthat final curly or if we reach the end of the file. The latter check ensures\nour compiler doesn't get stuck in an infinite loop if the user accidentally\nforgets the closing brace.\n\nThe tricky part with compiling a class declaration is that a class may declare\nany number of methods. Somehow the runtime needs to look up and bind all of\nthem. That would be a lot to pack into a single `OP_CLASS` instruction. Instead,\nthe bytecode we generate for a class declaration will split the process into a\n*series* of instructions. The compiler already emits\nan `OP_CLASS` instruction that creates a new empty ObjClass object. Then it\nemits instructions to store the class in a variable with its name.\n\n\n\nNow, for each method declaration, we emit a new `OP_METHOD` instruction that\nadds a single method to that class. When all of the `OP_METHOD` instructions\nhave executed, we're left with a fully formed class. While the user sees a class\ndeclaration as a single atomic operation, the VM implements it as a series of\nmutations.\n\nTo define a new method, the VM needs three things:\n\n1. The name of the method.\n\n1. The closure for the method body.\n\n1. The class to bind the method to.\n\nWe'll incrementally write the compiler code to see how those all get through to\nthe runtime, starting here:\n\n^code method\n\nLike `OP_GET_PROPERTY` and other instructions that need names at runtime, the\ncompiler adds the method name token's lexeme to the constant table, getting back\na table index. Then we emit an `OP_METHOD` instruction with that index as the\noperand. That's the name. Next is the method body:\n\n^code method-body (1 before, 1 after)\n\nWe use the same `function()` helper that we wrote for compiling function\ndeclarations. That utility function compiles the subsequent parameter list and\nfunction body. Then it emits the code to create an ObjClosure and leave it on\ntop of the stack. At runtime, the VM will find the closure there.\n\nLast is the class to bind the method to. Where can the VM find that?\nUnfortunately, by the time we reach the `OP_METHOD` instruction, we don't know\nwhere it is. It could be on the stack, if the user\ndeclared the class in a local scope. But a top-level class declaration ends up\nwith the ObjClass in the global variable table.\n\n\n\nFear not. The compiler does know the *name* of the class. We can capture it\nright after we consume its token.\n\n^code class-name (1 before, 1 after)\n\nAnd we know that no other declaration with that name could possibly shadow the\nclass. So we do the easy fix. Before we start binding methods, we emit whatever\ncode is necessary to load the class back on top of the stack.\n\n^code load-class (2 before, 1 after)\n\nRight before compiling the class body, we call\n`namedVariable()`. That helper function generates code to load a variable with\nthe given name onto the stack. Then we compile the methods.\n\n\n\nThis means that when we execute each `OP_METHOD` instruction, the stack has the\nmethod's closure on top with the class right under it. Once we've reached the\nend of the methods, we no longer need the class and tell the VM to pop it off\nthe stack.\n\n^code pop-class (1 before, 1 after)\n\nPutting all of that together, here is an example class declaration to throw at\nthe compiler:\n\n```lox\nclass Brunch {\n bacon() {}\n eggs() {}\n}\n```\n\nGiven that, here is what the compiler generates and how those instructions\naffect the stack at runtime:\n\n\"The\n\nAll that remains for us is to implement the runtime for that new `OP_METHOD`\ninstruction.\n\n### Executing method declarations\n\nFirst we define the opcode.\n\n^code method-op (1 before, 1 after)\n\nWe disassemble it like other instructions that have string constant operands.\n\n^code disassemble-method (2 before, 1 after)\n\nAnd over in the interpreter, we add a new case too.\n\n^code interpret-method (1 before, 1 after)\n\nThere, we read the method name from the constant table and pass it here:\n\n^code define-method\n\nThe method closure is on top of the stack, above the class it will be bound to.\nWe read those two stack slots and store the closure in the class's method table.\nThen we pop the closure since we're done with it.\n\nNote that we don't do any runtime type checking on the closure or class object.\nThat `AS_CLASS()` call is safe because the compiler itself generated the code\nthat causes the class to be in that stack slot. The VM trusts its own compiler.\n\n\n\nAfter the series of `OP_METHOD` instructions is done and the `OP_POP` has popped\nthe class, we will have a class with a nicely populated method table, ready to\nstart doing things. The next step is pulling those methods back out and using\nthem.\n\n## Method References\n\nMost of the time, methods are accessed and immediately called, leading to this\nfamiliar syntax:\n\n```lox\ninstance.method(argument);\n```\n\nBut remember, in Lox and some other languages, those two steps are distinct and\ncan be separated.\n\n```lox\nvar closure = instance.method;\nclosure(argument);\n```\n\nSince users *can* separate the operations, we have to implement them separately.\nThe first step is using our existing dotted property syntax to access a method\ndefined on the instance's class. That should return some kind of object that the\nuser can then call like a function.\n\nThe obvious approach is to look up the method in the class's method table and\nreturn the ObjClosure associated with that name. But we also need to remember\nthat when you access a method, `this` gets bound to the instance the method was\naccessed from. Here's the example from [when we added methods to jlox][jlox]:\n\n[jlox]: classes.html#methods-on-classes\n\n```lox\nclass Person {\n sayName() {\n print this.name;\n }\n}\n\nvar jane = Person();\njane.name = \"Jane\";\n\nvar method = jane.sayName;\nmethod(); // ?\n```\n\nThis should print \"Jane\", so the object returned by `.sayName` somehow needs to\nremember the instance it was accessed from when it later gets called. In jlox,\nwe implemented that \"memory\" using the interpreter's existing heap-allocated\nEnvironment class, which handled all variable storage.\n\nOur bytecode VM has a more complex architecture for storing state. [Local\nvariables and temporaries][locals] are on the stack, [globals][] are in a hash\ntable, and variables in closures use [upvalues][]. That necessitates a somewhat\nmore complex solution for tracking a method's receiver in clox, and a new\nruntime type.\n\n[locals]: local-variables.html#representing-local-variables\n[globals]: global-variables.html#variable-declarations\n[upvalues]: closures.html#upvalues\n\n### Bound methods\n\nWhen the user executes a method access, we'll find the closure for that method\nand wrap it in a new \"bound method\" object that tracks\nthe instance that the method was accessed from. This bound object can be called\nlater like a function. When invoked, the VM will do some shenanigans to wire up\n`this` to point to the receiver inside the method's body.\n\n\n\nHere's the new object type:\n\n^code obj-bound-method (2 before, 1 after)\n\nIt wraps the receiver and the method closure together. The receiver's type is\nValue even though methods can be called only on ObjInstances. Since the VM\ndoesn't care what kind of receiver it has anyway, using Value means we don't\nhave to keep converting the pointer back to a Value when it gets passed to more\ngeneral functions.\n\nThe new struct implies the usual boilerplate you're used to by now. A new case\nin the object type enum:\n\n^code obj-type-bound-method (1 before, 1 after)\n\nA macro to check a value's type:\n\n^code is-bound-method (2 before, 1 after)\n\nAnother macro to cast the value to an ObjBoundMethod pointer:\n\n^code as-bound-method (2 before, 1 after)\n\nA function to create a new ObjBoundMethod:\n\n^code new-bound-method-h (2 before, 1 after)\n\nAnd an implementation of that function here:\n\n^code new-bound-method\n\nThe constructor-like function simply stores the given closure and receiver. When\nthe bound method is no longer needed, we free it.\n\n^code free-bound-method (1 before, 1 after)\n\nThe bound method has a couple of references, but it doesn't *own* them, so it\nfrees nothing but itself. However, those references do get traced by the garbage\ncollector.\n\n^code blacken-bound-method (1 before, 1 after)\n\nThis ensures that a handle to a method keeps the\nreceiver around in memory so that `this` can still find the object when you\ninvoke the handle later. We also trace the method closure.\n\n\n\nThe last operation all objects support is printing.\n\n^code print-bound-method (1 before, 1 after)\n\nA bound method prints exactly the same way as a function. From the user's\nperspective, a bound method *is* a function. It's an object they can call. We\ndon't expose that the VM implements bound methods using a different object type.\n\n\n\nPut on your party hat because we just reached a little\nmilestone. ObjBoundMethod is the very last runtime type to add to clox. You've\nwritten your last `IS_` and `AS_` macros. We're only a few chapters from the end\nof the book, and we're getting close to a complete VM.\n\n### Accessing methods\n\nLet's get our new object type doing something. Methods are accessed using the\nsame \"dot\" property syntax we implemented in the last chapter. The compiler\nalready parses the right expressions and emits `OP_GET_PROPERTY` instructions\nfor them. The only changes we need to make are in the runtime.\n\nWhen a property access instruction executes, the instance is on top of the\nstack. The instruction's job is to find a field or method with the given name\nand replace the top of the stack with the accessed property.\n\nThe interpreter already handles fields, so we simply extend the\n`OP_GET_PROPERTY` case with another section.\n\n^code get-method (5 before, 1 after)\n\nWe insert this after the code to look up a field on the receiver instance.\nFields take priority over and shadow methods, so we look for a field first. If\nthe instance does not have a field with the given property name, then the name\nmay refer to a method.\n\nWe take the instance's class and pass it to a new `bindMethod()` helper. If that\nfunction finds a method, it places the method on the stack and returns `true`.\nOtherwise it returns `false` to indicate a method with that name couldn't be\nfound. Since the name also wasn't a field, that means we have a runtime error,\nwhich aborts the interpreter.\n\nHere is the good stuff:\n\n^code bind-method\n\nFirst we look for a method with the given name in the class's method table. If\nwe don't find one, we report a runtime error and bail out. Otherwise, we take\nthe method and wrap it in a new ObjBoundMethod. We grab the receiver from its\nhome on top of the stack. Finally, we pop the instance and replace the top of\nthe stack with the bound method.\n\nFor example:\n\n```lox\nclass Brunch {\n eggs() {}\n}\n\nvar brunch = Brunch();\nvar eggs = brunch.eggs;\n```\n\nHere is what happens when the VM executes the `bindMethod()` call for the\n`brunch.eggs` expression:\n\n\"The\n\nThat's a lot of machinery under the hood, but from the user's perspective, they\nsimply get a function that they can call.\n\n### Calling methods\n\nUsers can declare methods on classes, access them on instances, and get bound\nmethods onto the stack. They just can't *do* anything\nuseful with those bound method objects. The operation we're missing is calling\nthem. Calls are implemented in `callValue()`, so we add a case there for the new\nobject type.\n\n\n\n^code call-bound-method (1 before, 1 after)\n\nWe pull the raw closure back out of the ObjBoundMethod and use the existing\n`call()` helper to begin an invocation of that closure by pushing a CallFrame\nfor it onto the call stack. That's all it takes to be able to run this Lox\nprogram:\n\n```lox\nclass Scone {\n topping(first, second) {\n print \"scone with \" + first + \" and \" + second;\n }\n}\n\nvar scone = Scone();\nscone.topping(\"berries\", \"cream\");\n```\n\nThat's three big steps. We can declare, access, and invoke methods. But\nsomething is missing. We went to all that trouble to wrap the method closure in\nan object that binds the receiver, but when we invoke the method, we don't use\nthat receiver at all.\n\n## This\n\nThe reason bound methods need to keep hold of the receiver is so that it can be\naccessed inside the body of the method. Lox exposes a method's receiver through\n`this` expressions. It's time for some new syntax. The lexer already treats\n`this` as a special token type, so the first step is wiring that token up in the\nparse table.\n\n^code table-this (1 before, 1 after)\n\n\n\nWhen the parser encounters a `this` in prefix position, it dispatches to a new\nparser function.\n\n^code this\n\nWe'll apply the same implementation technique for `this` in clox that we used in\njlox. We treat `this` as a lexically scoped local variable whose value gets\nmagically initialized. Compiling it like a local variable means we get a lot of\nbehavior for free. In particular, closures inside a method that reference `this`\nwill do the right thing and capture the receiver in an upvalue.\n\nWhen the parser function is called, the `this` token has just been consumed and\nis stored as the previous token. We call our existing `variable()` function\nwhich compiles identifier expressions as variable accesses. It takes a single\nBoolean parameter for whether the compiler should look for a following `=`\noperator and parse a setter. You can't assign to `this`, so we pass `false` to\ndisallow that.\n\nThe `variable()` function doesn't care that `this` has its own token type and\nisn't an identifier. It is happy to treat the lexeme \"this\" as if it were a\nvariable name and then look it up using the existing scope resolution machinery.\nRight now, that lookup will fail because we never declared a variable whose name\nis \"this\". It's time to think about where the receiver should live in memory.\n\nAt least until they get captured by closures, clox stores every local variable\non the VM's stack. The compiler keeps track of which slots in the function's\nstack window are owned by which local variables. If you recall, the compiler\nsets aside stack slot zero by declaring a local variable whose name is an empty\nstring.\n\nFor function calls, that slot ends up holding the function being called. Since\nthe slot has no name, the function body never accesses it. You can guess where\nthis is going. For *method* calls, we can repurpose that slot to store the\nreceiver. Slot zero will store the instance that `this` is bound to. In order to\ncompile `this` expressions, the compiler simply needs to give the correct name\nto that local variable.\n\n^code slot-zero (1 before, 1 after)\n\nWe want to do this only for methods. Function declarations don't have a `this`.\nAnd, in fact, they *must not* declare a variable named \"this\", so that if you\nwrite a `this` expression inside a function declaration which is itself inside a\nmethod, the `this` correctly resolves to the outer method's receiver.\n\n```lox\nclass Nested {\n method() {\n fun function() {\n print this;\n }\n\n function();\n }\n}\n\nNested().method();\n```\n\nThis program should print \"Nested instance\". To decide what name to give to\nlocal slot zero, the compiler needs to know whether it's compiling a function or\nmethod declaration, so we add a new case to our FunctionType enum to distinguish\nmethods.\n\n^code method-type-enum (1 before, 1 after)\n\nWhen we compile a method, we use that type.\n\n^code method-type (2 before, 1 after)\n\nNow we can correctly compile references to the special \"this\" variable, and the\ncompiler will emit the right `OP_GET_LOCAL` instructions to access it. Closures\ncan even capture `this` and store the receiver in upvalues. Pretty cool.\n\nExcept that at runtime, the receiver isn't actually *in* slot zero. The\ninterpreter isn't holding up its end of the bargain yet. Here is the fix:\n\n^code store-receiver (2 before, 2 after)\n\nWhen a method is called, the top of the stack contains all of the arguments, and\nthen just under those is the closure of the called method. That's where slot\nzero in the new CallFrame will be. This line of code inserts the receiver into\nthat slot. For example, given a method call like this:\n\n```lox\nscone.topping(\"berries\", \"cream\");\n```\n\nWe calculate the slot to store the receiver like so:\n\n\"Skipping\n\nThe `-argCount` skips past the arguments and the `- 1` adjusts for the fact that\n`stackTop` points just *past* the last used stack slot.\n\n### Misusing this\n\nOur VM now supports users *correctly* using `this`, but we also need to make\nsure it properly handles users *mis*using `this`. Lox says it is a compile\nerror for a `this` expression to appear outside of the body of a method. These\ntwo wrong uses should be caught by the compiler:\n\n```lox\nprint this; // At top level.\n\nfun notMethod() {\n print this; // In a function.\n}\n```\n\nSo how does the compiler know if it's inside a method? The obvious answer is to\nlook at the FunctionType of the current Compiler. We did just add an enum case\nthere to treat methods specially. However, that wouldn't correctly handle code\nlike the earlier example where you are inside a function which is, itself,\nnested inside a method.\n\nWe could try to resolve \"this\" and then report an error if it wasn't found in\nany of the surrounding lexical scopes. That would work, but would require us to\nshuffle around a bunch of code, since right now the code for resolving a\nvariable implicitly considers it a global access if no declaration is found.\n\nIn the next chapter, we will need information about the nearest enclosing class.\nIf we had that, we could use it here to determine if we are inside a method. So\nwe may as well make our future selves' lives a little easier and put that\nmachinery in place now.\n\n^code current-class (1 before, 2 after)\n\nThis module variable points to a struct representing the current, innermost\nclass being compiled. The new type looks like this:\n\n^code class-compiler-struct (1 before, 2 after)\n\nRight now we store only a pointer to the ClassCompiler for the enclosing class,\nif any. Nesting a class declaration inside a method in some other class is an\nuncommon thing to do, but Lox supports it. Just like the Compiler struct, this\nmeans ClassCompiler forms a linked list from the current innermost class being\ncompiled out through all of the enclosing classes.\n\nIf we aren't inside any class declaration at all, the module variable\n`currentClass` is `NULL`. When the compiler begins compiling a class, it pushes\na new ClassCompiler onto that implicit linked stack.\n\n^code create-class-compiler (2 before, 1 after)\n\nThe memory for the ClassCompiler struct lives right on the C stack, a handy\ncapability we get by writing our compiler using recursive descent. At the end of\nthe class body, we pop that compiler off the stack and restore the enclosing\none.\n\n^code pop-enclosing (1 before, 1 after)\n\nWhen an outermost class body ends, `enclosing` will be `NULL`, so this resets\n`currentClass` to `NULL`. Thus, to see if we are inside a class -- and therefore\ninside a method -- we simply check that module variable.\n\n^code this-outside-class (1 before, 1 after)\n\nWith that, `this` outside of a class is correctly forbidden. Now our methods\nreally feel like *methods* in the object-oriented sense. Accessing the receiver\nlets them affect the instance you called the method on. We're getting there!\n\n## Instance Initializers\n\nThe reason object-oriented languages tie state and behavior together -- one of\nthe core tenets of the paradigm -- is to ensure that objects are always in a\nvalid, meaningful state. When the only way to touch an object's state is through its methods, the methods can make sure nothing\ngoes awry. But that presumes the object is *already* in a proper state. What\nabout when it's first created?\n\n\n\nObject-oriented languages ensure that brand new objects are properly set up\nthrough constructors, which both produce a new instance and initialize its\nstate. In Lox, the runtime allocates new raw instances, and a class may declare\nan initializer to set up any fields. Initializers work mostly like normal\nmethods, with a few tweaks:\n\n1. The runtime automatically invokes the initializer method whenever an\n instance of a class is created.\n\n2. The caller that constructs an instance always gets the instance back after the initializer finishes, regardless of what\n the initializer function itself returns. The initializer method doesn't need\n to explicitly return `this`.\n\n3. In fact, an initializer is *prohibited* from returning any value at all\n since the value would never be seen anyway.\n\n\n\nNow that we support methods, to add initializers, we merely need to implement\nthose three special rules. We'll go in order.\n\n### Invoking initializers\n\nFirst, automatically calling `init()` on new instances:\n\n^code call-init (1 before, 1 after)\n\nAfter the runtime allocates the new instance, we look for an `init()` method on\nthe class. If we find one, we initiate a call to it. This pushes a new CallFrame\nfor the initializer's closure. Say we run this program:\n\n```lox\nclass Brunch {\n init(food, drink) {}\n}\n\nBrunch(\"eggs\", \"coffee\");\n```\n\nWhen the VM executes the call to `Brunch()`, it goes like this:\n\n\"The\n\nAny arguments passed to the class when we called it are still sitting on the\nstack above the instance. The new CallFrame for the `init()` method shares that\nstack window, so those arguments implicitly get forwarded to the initializer.\n\nLox doesn't require a class to define an initializer. If omitted, the runtime\nsimply returns the new uninitialized instance. However, if there is no `init()`\nmethod, then it doesn't make any sense to pass arguments to the class when\ncreating the instance. We make that an error.\n\n^code no-init-arity-error (1 before, 1 after)\n\nWhen the class *does* provide an initializer, we also need to ensure that the\nnumber of arguments passed matches the initializer's arity. Fortunately, the\n`call()` helper does that for us already.\n\nTo call the initializer, the runtime looks up the `init()` method by name. We\nwant that to be fast since it happens every time an instance is constructed.\nThat means it would be good to take advantage of the string interning we've\nalready implemented. To do that, the VM creates an ObjString for \"init\" and\nreuses it. The string lives right in the VM struct.\n\n^code vm-init-string (1 before, 1 after)\n\nWe create and intern the string when the VM boots up.\n\n^code init-init-string (1 before, 2 after)\n\nWe want it to stick around, so the GC considers it a root.\n\n^code mark-init-string (1 before, 1 after)\n\nLook carefully. See any bug waiting to happen? No? It's a subtle one. The\ngarbage collector now reads `vm.initString`. That field is initialized from the\nresult of calling `copyString()`. But copying a string allocates memory, which\ncan trigger a GC. If the collector ran at just the wrong time, it would read\n`vm.initString` before it had been initialized. So, first we zero the field out.\n\n^code null-init-string (2 before, 2 after)\n\nWe clear the pointer when the VM shuts down since the next line will free it.\n\n^code clear-init-string (1 before, 1 after)\n\nOK, that lets us call initializers.\n\n### Initializer return values\n\nThe next step is ensuring that constructing an instance of a class with an\ninitializer always returns the new instance, and not `nil` or whatever the body\nof the initializer returns. Right now, if a class defines an initializer, then\nwhen an instance is constructed, the VM pushes a call to that initializer onto\nthe CallFrame stack. Then it just keeps on trucking.\n\nThe user's invocation on the class to create the instance will complete whenever\nthat initializer method returns, and will leave on the stack whatever value the\ninitializer puts there. That means that unless the user takes care to put\n`return this;` at the end of the initializer, no instance will come out. Not\nvery helpful.\n\nTo fix this, whenever the front end compiles an initializer method, it will emit\ndifferent bytecode at the end of the body to return `this` from the method\ninstead of the usual implicit `nil` most functions return. In order to do\n*that*, the compiler needs to actually know when it is compiling an initializer.\nWe detect that by checking to see if the name of the method we're compiling is\n\"init\".\n\n^code initializer-name (1 before, 1 after)\n\nWe define a new function type to distinguish initializers from other methods.\n\n^code initializer-type-enum (1 before, 1 after)\n\nWhenever the compiler emits the implicit return at the end of a body, we check\nthe type to decide whether to insert the initializer-specific behavior.\n\n^code return-this (1 before, 1 after)\n\nIn an initializer, instead of pushing `nil` onto the stack before returning,\nwe load slot zero, which contains the instance. This `emitReturn()` function is\nalso called when compiling a `return` statement without a value, so this also\ncorrectly handles cases where the user does an early return inside the\ninitializer.\n\n### Incorrect returns in initializers\n\nThe last step, the last item in our list of special features of initializers, is\nmaking it an error to try to return anything *else* from an initializer. Now\nthat the compiler tracks the method type, this is straightforward.\n\n^code return-from-init (3 before, 1 after)\n\nWe report an error if a `return` statement in an initializer has a value. We\nstill go ahead and compile the value afterwards so that the compiler doesn't get\nconfused by the trailing expression and report a bunch of cascaded errors.\n\nAside from inheritance, which we'll get to [soon][super], we now have a\nfairly full-featured class system working in clox.\n\n```lox\nclass CoffeeMaker {\n init(coffee) {\n this.coffee = coffee;\n }\n\n brew() {\n print \"Enjoy your cup of \" + this.coffee;\n\n // No reusing the grounds!\n this.coffee = nil;\n }\n}\n\nvar maker = CoffeeMaker(\"coffee and chicory\");\nmaker.brew();\n```\n\nPretty fancy for a C program that would fit on an old floppy disk.\n\n\n\n## Optimized Invocations\n\nOur VM correctly implements the language's semantics for method calls and\ninitializers. We could stop here. But the main reason we are building an entire\nsecond implementation of Lox from scratch is to execute faster than our old Java\ninterpreter. Right now, method calls even in clox are slow.\n\nLox's semantics define a method invocation as two operations -- accessing the\nmethod and then calling the result. Our VM must support those as separate\noperations because the user *can* separate them. You can access a method without\ncalling it and then invoke the bound method later. Nothing we've implemented so\nfar is unnecessary.\n\nBut *always* executing those as separate operations has a significant cost.\nEvery single time a Lox program accesses and invokes a method, the runtime\nheap allocates a new ObjBoundMethod, initializes its fields, then pulls them\nright back out. Later, the GC has to spend time freeing all of those ephemeral\nbound methods.\n\nMost of the time, a Lox program accesses a method and then immediately calls it.\nThe bound method is created by one bytecode instruction and then consumed by the\nvery next one. In fact, it's so immediate that the compiler can even textually\n*see* that it's happening -- a dotted property access followed by an opening\nparenthesis is most likely a method call.\n\nSince we can recognize this pair of operations at compile time, we have the\nopportunity to emit a new, special instruction that\nperforms an optimized method call.\n\nWe start in the function that compiles dotted property expressions.\n\n\n\n^code parse-call (3 before, 1 after)\n\nAfter the compiler has parsed the property name, we look for a left parenthesis.\nIf we match one, we switch to a new code path. There, we compile the argument\nlist exactly like we do when compiling a call expression. Then we emit a single\nnew `OP_INVOKE` instruction. It takes two operands:\n\n1. The index of the property name in the constant table.\n\n2. The number of arguments passed to the method.\n\nIn other words, this single instruction combines the operands of the\n`OP_GET_PROPERTY` and `OP_CALL` instructions it replaces, in that order. It\nreally is a fusion of those two instructions. Let's define it.\n\n^code invoke-op (1 before, 1 after)\n\nAnd add it to the disassembler:\n\n^code disassemble-invoke (2 before, 1 after)\n\nThis is a new, special instruction format, so it needs a little custom\ndisassembly logic.\n\n^code invoke-instruction\n\nWe read the two operands and then print out both the method name and the\nargument count. Over in the interpreter's bytecode dispatch loop is where the\nreal action begins.\n\n^code interpret-invoke (1 before, 1 after)\n\nMost of the work happens in `invoke()`, which we'll get to. Here, we look up the\nmethod name from the first operand and then read the argument count operand.\nThen we hand off to `invoke()` to do the heavy lifting. That function returns\n`true` if the invocation succeeds. As usual, a `false` return means a runtime\nerror occurred. We check for that here and abort the interpreter if disaster has\nstruck.\n\nFinally, assuming the invocation succeeded, then there is a new CallFrame on the\nstack, so we refresh our cached copy of the current frame in `frame`.\n\nThe interesting work happens here:\n\n^code invoke\n\nFirst we grab the receiver off the stack. The arguments passed to the method are\nabove it on the stack, so we peek that many slots down. Then it's a simple\nmatter to cast the object to an instance and invoke the method on it.\n\nThat does assume the object *is* an instance. As with `OP_GET_PROPERTY`\ninstructions, we also need to handle the case where a user incorrectly tries to\ncall a method on a value of the wrong type.\n\n^code invoke-check-type (1 before, 1 after)\n\nThat's a runtime error, so we report that and bail\nout. Otherwise, we get the instance's class and jump over to this other new\nutility function:\n\n\n\n^code invoke-from-class\n\nThis function combines the logic of how the VM implements `OP_GET_PROPERTY` and\n`OP_CALL` instructions, in that order. First we look up the method by name in\nthe class's method table. If we don't find one, we report that runtime error and\nexit.\n\nOtherwise, we take the method's closure and push a call to it onto the CallFrame\nstack. We don't need to heap allocate and initialize an ObjBoundMethod. In fact,\nwe don't even need to juggle anything on the stack.\nThe receiver and method arguments are already right where they need to be.\n\n\n\nIf you fire up the VM and run a little program that calls methods now, you\nshould see the exact same behavior as before. But, if we did our job right, the\n*performance* should be much improved. I wrote a little microbenchmark that\ndoes a batch of 10,000 method calls. Then it tests how many of these batches it\ncan execute in 10 seconds. On my computer, without the new `OP_INVOKE`\ninstruction, it got through 1,089 batches. With this new optimization, it\nfinished 8,324 batches in the same time. That's *7.6 times faster*, which is a\nhuge improvement when it comes to programming language optimization.\n\n\n\n\n\n\"Bar\n\n### Invoking fields\n\nThe fundamental creed of optimization is: \"Thou shalt not break correctness.\"\nUsers like it when a language implementation gives\nthem an answer faster, but only if it's the *right* answer. Alas, our\nimplementation of faster method invocations fails to uphold that principle:\n\n```lox\nclass Oops {\n init() {\n fun f() {\n print \"not a method\";\n }\n\n this.field = f;\n }\n}\n\nvar oops = Oops();\noops.field();\n```\n\nThe last line looks like a method call. The compiler thinks that it is and\ndutifully emits an `OP_INVOKE` instruction for it. However, it's not. What is\nactually happening is a *field* access that returns a function which then gets\ncalled. Right now, instead of executing that correctly, our VM reports a runtime\nerror when it can't find a method named \"field\".\n\n\n\nEarlier, when we implemented `OP_GET_PROPERTY`, we handled both field and method\naccesses. To squash this new bug, we need to do the same thing for `OP_INVOKE`.\n\n^code invoke-field (1 before, 1 after)\n\nPretty simple fix. Before looking up a method on the instance's class, we look\nfor a field with the same name. If we find a field, then we store it on the\nstack in place of the receiver, *under* the argument list. This is how\n`OP_GET_PROPERTY` behaves since the latter instruction executes before a\nsubsequent parenthesized list of arguments has been evaluated.\n\nThen we try to call that field's value like the callable that it hopefully is.\nThe `callValue()` helper will check the value's type and call it as appropriate\nor report a runtime error if the field's value isn't a callable type like a\nclosure.\n\nThat's all it takes to make our optimization fully safe. We do sacrifice a\nlittle performance, unfortunately. But that's the price you have to pay\nsometimes. You occasionally get frustrated by optimizations you *could* do if\nonly the language wouldn't allow some annoying corner case. But, as language\nimplementers, we have to play the game we're given.\n\n\n\nThe code we wrote here follows a typical pattern in optimization:\n\n1. Recognize a common operation or sequence of operations that is performance\n critical. In this case, it is a method access followed by a call.\n\n2. Add an optimized implementation of that pattern. That's our `OP_INVOKE`\n instruction.\n\n3. Guard the optimized code with some conditional logic that validates that the\n pattern actually applies. If it does, stay on the fast path. Otherwise, fall\n back to a slower but more robust unoptimized behavior. Here, that means\n checking that we are actually calling a method and not accessing a field.\n\nAs your language work moves from getting the implementation working *at all* to\ngetting it to work *faster*, you will find yourself spending more and more\ntime looking for patterns like this and adding guarded optimizations for them.\nFull-time VM engineers spend much of their careers in this loop.\n\nBut we can stop here for now. With this, clox now supports most of the features\nof an object-oriented programming language, and with respectable performance.\n\n
\n\n## Challenges\n\n1. The hash table lookup to find a class's `init()` method is constant time,\n but still fairly slow. Implement something faster. Write a benchmark and\n measure the performance difference.\n\n1. In a dynamically typed language like Lox, a single callsite may invoke a\n variety of methods on a number of classes throughout a program's execution.\n Even so, in practice, most of the time a callsite ends up calling the exact\n same method on the exact same class for the duration of the run. Most calls\n are actually not polymorphic even if the language says they can be.\n\n How do advanced language implementations optimize based on that observation?\n\n1. When interpreting an `OP_INVOKE` instruction, the VM has to do two hash\n table lookups. First, it looks for a field that could shadow a method, and\n only if that fails does it look for a method. The former check is rarely\n useful -- most fields do not contain functions. But it is *necessary*\n because the language says fields and methods are accessed using the same\n syntax, and fields shadow methods.\n\n That is a language *choice* that affects the performance of our\n implementation. Was it the right choice? If Lox were your language, what\n would you do?\n\n
\n\n
\n\n## Design Note: Novelty Budget\n\nI still remember the first time I wrote a tiny BASIC program on a TRS-80 and\nmade a computer do something it hadn't done before. It felt like a superpower.\nThe first time I cobbled together just enough of a parser and interpreter to let\nme write a tiny program in *my own language* that made a computer do a thing was\nlike some sort of higher-order meta-superpower. It was and remains a wonderful\nfeeling.\n\nI realized I could design a language that looked and behaved however I chose. It\nwas like I'd been going to a private school that required uniforms my whole life\nand then one day transferred to a public school where I could wear whatever I\nwanted. I don't need to use curly braces for blocks? I can use something other\nthan an equals sign for assignment? I can do objects without classes? Multiple\ninheritance *and* multimethods? A dynamic language that overloads statically, by\narity?\n\nNaturally, I took that freedom and ran with it. I made the weirdest, most\narbitrary language design decisions. Apostrophes for generics. No commas between\narguments. Overload resolution that can fail at runtime. I did things\ndifferently just for difference's sake.\n\nThis is a very fun experience that I highly recommend. We need more weird,\navant-garde programming languages. I want to see more art languages. I still\nmake oddball toy languages for fun sometimes.\n\n*However*, if your goal is success where \"success\" is defined as a large number\nof users, then your priorities must be different. In that case, your primary\ngoal is to have your language loaded into the brains of as many people as\npossible. That's *really hard*. It takes a lot of human effort to move a\nlanguage's syntax and semantics from a computer into trillions of neurons.\n\nProgrammers are naturally conservative with their time and cautious about what\nlanguages are worth uploading into their wetware. They don't want to waste their\ntime on a language that ends up not being useful to them. As a language\ndesigner, your goal is thus to give them as much language power as you can with\nas little required learning as possible.\n\nOne natural approach is *simplicity*. The fewer concepts and features your\nlanguage has, the less total volume of stuff there is to learn. This is one of\nthe reasons minimal scripting languages often find\nsuccess even though they aren't as powerful as the big industrial languages --\nthey are easier to get started with, and once they are in someone's brain, the\nuser wants to keep using them.\n\n\n\nThe problem with simplicity is that simply cutting features often sacrifices\npower and expressiveness. There is an art to finding features that punch above\ntheir weight, but often minimal languages simply do less.\n\nThere is another path that avoids much of that problem. The trick is to realize\nthat a user doesn't have to load your entire language into their head, *just the\npart they don't already have in there*. As I mentioned in an [earlier design\nnote][note], learning is about transferring the *delta* between what they\nalready know and what they need to know.\n\n[note]: parsing-expressions.html#design-note\n\nMany potential users of your language already know some other programming\nlanguage. Any features your language shares with that language are essentially\n\"free\" when it comes to learning. It's already in their head, they just have to\nrecognize that your language does the same thing.\n\nIn other words, *familiarity* is another key tool to lower the adoption cost of\nyour language. Of course, if you fully maximize that attribute, the end result\nis a language that is completely identical to some existing one. That's not a\nrecipe for success, because at that point there's no incentive for users to\nswitch to your language at all.\n\nSo you do need to provide some compelling differences. Some things your language\ncan do that other languages can't, or at least can't do as well. I believe this\nis one of the fundamental balancing acts of language design: similarity to other\nlanguages lowers learning cost, while divergence raises the compelling\nadvantages.\n\nI think of this balancing act in terms of a **novelty\nbudget**, or as Steve Klabnik calls it, a \"[strangeness budget][]\". Users\nhave a low threshold for the total amount of new stuff they are willing to\naccept to learn a new language. Exceed that, and they won't show up.\n\n[strangeness budget]: https://words.steveklabnik.com/the-language-strangeness-budget\n\n\n\nAnytime you add something new to your language that other languages don't have,\nor anytime your language does something other languages do in a different way,\nyou spend some of that budget. That's OK -- you *need* to spend it to make your\nlanguage compelling. But your goal is to spend it *wisely*. For each feature or\ndifference, ask yourself how much compelling power it adds to your language and\nthen evaluate critically whether it pays its way. Is the change so valuable that\nit is worth blowing some of your novelty budget?\n\nIn practice, I find this means that you end up being pretty conservative with\nsyntax and more adventurous with semantics. As fun as it is to put on a new\nchange of clothes, swapping out curly braces with some other block delimiter is\nvery unlikely to add much real power to the language, but it does spend some\nnovelty. It's hard for syntax differences to carry their weight.\n\nOn the other hand, new semantics can significantly increase the power of the\nlanguage. Multimethods, mixins, traits, reflection, dependent types, runtime\nmetaprogramming, etc. can radically level up what a user can do with the\nlanguage.\n\nAlas, being conservative like this is not as fun as just changing everything.\nBut it's up to you to decide whether you want to chase mainstream success or not\nin the first place. We don't all need to be radio-friendly pop bands. If you\nwant your language to be like free jazz or drone metal and are happy with the\nproportionally smaller (but likely more devoted) audience size, go for it.\n\n
\n" }, { "path": "book/optimization.md", "content": "> The evening's the best part of the day. You've done your day's work. Now you\n> can put your feet up and enjoy it.\n>\n> Kazuo Ishiguro, The Remains of the Day\n\nIf I still lived in New Orleans, I'd call this chapter a *lagniappe*, a little\nsomething extra given for free to a customer. You've got a whole book and a\ncomplete virtual machine already, but I want you to have some more fun hacking\non clox. This time, we're going for pure performance. We'll apply two very\ndifferent optimizations to our virtual machine. In the process, you'll get a\nfeel for measuring and improving the performance of a language implementation --\nor any program, really.\n\n## Measuring Performance\n\n**Optimization** means taking a working application and improving its\nperformance. An optimized program does the same thing, it just takes less\nresources to do so. The resource we usually think of when optimizing is runtime\nspeed, but it can also be important to reduce memory usage, startup time,\npersistent storage size, or network bandwidth. All physical resources have some\ncost -- even if the cost is mostly in wasted human time -- so optimization work\noften pays off.\n\nThere was a time in the early days of computing that a skilled programmer could\nhold the entire hardware architecture and compiler pipeline in their head and\nunderstand a program's performance just by thinking real hard. Those days are\nlong gone, separated from the present by microcode, cache lines, branch\nprediction, deep compiler pipelines, and mammoth instruction sets. We like to\npretend C is a \"low-level\" language, but the stack of technology between\n\n```c\nprintf(\"Hello, world!\");\n```\n\nand a greeting appearing on screen is now perilously tall.\n\nOptimization today is an empirical science. Our program is a border collie\nsprinting through the hardware's obstacle course. If we want her to reach the\nend faster, we can't just sit and ruminate on canine physiology until\nenlightenment strikes. Instead, we need to *observe* her performance, see where\nshe stumbles, and then find faster paths for her to take.\n\nMuch like agility training is particular to one dog and one obstacle course, we\ncan't assume that our virtual machine optimizations will make *all* Lox programs\nrun faster on *all* hardware. Different Lox programs stress different areas of\nthe VM, and different architectures have their own strengths and weaknesses.\n\n### Benchmarks\n\nWhen we add new functionality, we validate correctness by writing tests -- Lox\nprograms that use a feature and validate the VM's behavior. Tests pin down\nsemantics and ensure we don't break existing features when we add new ones. We\nhave similar needs when it comes to performance:\n\n1. How do we validate that an optimization *does* improve performance, and by\n how much?\n\n2. How do we ensure that other unrelated changes don't *regress* performance?\n\nThe Lox programs we write to accomplish those goals are **benchmarks**. These\nare carefully crafted programs that stress some part of the language\nimplementation. They measure not *what* the program does, but how *long* it takes to do it.\n\n\n\nBy measuring the performance of a benchmark before and after a change, you can\nsee what your change does. When you land an optimization, all of the tests\nshould behave exactly the same as they did before, but hopefully the benchmarks\nrun faster.\n\nOnce you have an entire *suite* of benchmarks, you can\nmeasure not just *that* an optimization changes performance, but on which\n*kinds* of code. Often you'll find that some benchmarks get faster while others\nget slower. Then you have to make hard decisions about what kinds of code your\nlanguage implementation optimizes for.\n\nThe suite of benchmarks you choose to write is a key part of that decision. In\nthe same way that your tests encode your choices around what correct behavior\nlooks like, your benchmarks are the embodiment of your priorities when it comes\nto performance. They will guide which optimizations you implement, so choose\nyour benchmarks carefully, and don't forget to periodically reflect on whether\nthey are helping you reach your larger goals.\n\n\n\nBenchmarking is a subtle art. Like tests, you need to balance not overfitting to\nyour implementation while ensuring that the benchmark does actually tickle the\ncode paths that you care about. When you measure performance, you need to\ncompensate for variance caused by CPU throttling, caching, and other weird\nhardware and operating system quirks. I won't give you a whole sermon here,\nbut treat benchmarking as its own skill that improves with practice.\n\n### Profiling\n\nOK, so you've got a few benchmarks now. You want to make them go faster. Now\nwhat? First of all, let's assume you've done all the obvious, easy work. You are\nusing the right algorithms and data structures -- or, at least, you aren't using\nones that are aggressively wrong. I don't consider using a hash table instead of\na linear search through a huge unsorted array \"optimization\" so much as \"good\nsoftware engineering\".\n\nSince the hardware is too complex to reason about our program's performance from\nfirst principles, we have to go out into the field. That means *profiling*. A\n**profiler**, if you've never used one, is a tool that runs your program and tracks hardware resource use as the code\nexecutes. Simple ones show you how much time was spent in each function in your\nprogram. Sophisticated ones log data cache misses, instruction cache misses,\nbranch mispredictions, memory allocations, and all sorts of other metrics.\n\n\n\nThere are many profilers out there for various operating systems and languages.\nOn whatever platform you program, it's worth getting familiar with a decent\nprofiler. You don't need to be a master. I have learned things within minutes of\nthrowing a program at a profiler that would have taken me *days* to discover on\nmy own through trial and error. Profilers are wonderful, magical tools.\n\n## Faster Hash Table Probing\n\nEnough pontificating, let's get some performance charts going up and to the\nright. The first optimization we'll do, it turns out, is about the *tiniest*\npossible change we could make to our VM.\n\nWhen I first got the bytecode virtual machine that clox is descended from\nworking, I did what any self-respecting VM hacker would do. I cobbled together a\ncouple of benchmarks, fired up a profiler, and ran those scripts through my\ninterpreter. In a dynamically typed language like Lox, a large fraction of user\ncode is field accesses and method calls, so one of my benchmarks looked\nsomething like this:\n\n```lox\nclass Zoo {\n init() {\n this.aardvark = 1;\n this.baboon = 1;\n this.cat = 1;\n this.donkey = 1;\n this.elephant = 1;\n this.fox = 1;\n }\n ant() { return this.aardvark; }\n banana() { return this.baboon; }\n tuna() { return this.cat; }\n hay() { return this.donkey; }\n grass() { return this.elephant; }\n mouse() { return this.fox; }\n}\n\nvar zoo = Zoo();\nvar sum = 0;\nvar start = clock();\nwhile (sum < 100000000) {\n sum = sum + zoo.ant()\n + zoo.banana()\n + zoo.tuna()\n + zoo.hay()\n + zoo.grass()\n + zoo.mouse();\n}\n\nprint clock() - start;\nprint sum;\n```\n\n\n\nIf you've never seen a benchmark before, this might seem ludicrous. *What* is\ngoing on here? The program itself doesn't intend to do\nanything useful. What it does do is call a bunch of methods and access a bunch\nof fields since those are the parts of the language we're interested in. Fields\nand methods live in hash tables, so it takes care to populate at least a *few* interesting keys in those tables. That is all wrapped\nin a big loop to ensure our profiler has enough execution time to dig in and see\nwhere the cycles are going.\n\n\n\nBefore I tell you what my profiler showed me, spend a minute taking a few\nguesses. Where in clox's codebase do you think the VM spent most of its time? Is\nthere any code we've written in previous chapters that you suspect is\nparticularly slow?\n\nHere's what I found: Naturally, the function with the greatest inclusive time is\n`run()`. (**Inclusive time** means the total time spent in some function and all\nother functions it calls -- the total time between when you enter the function\nand when it returns.) Since `run()` is the main bytecode execution loop, it\ndrives everything.\n\nInside `run()`, there are small chunks of time sprinkled in various cases in the\nbytecode switch for common instructions like `OP_POP`, `OP_RETURN`, and\n`OP_ADD`. The big heavy instructions are `OP_GET_GLOBAL` with 17% of the\nexecution time, `OP_GET_PROPERTY` at 12%, and `OP_INVOKE` which takes a whopping\n42% of the total running time.\n\nSo we've got three hotspots to optimize? Actually, no. Because it turns out\nthose three instructions spend almost all of their time inside calls to the same\nfunction: `tableGet()`. That function claims a whole 72% of the execution time\n(again, inclusive). Now, in a dynamically typed language, we expect to spend a\nfair bit of time looking stuff up in hash tables -- it's sort of the price of\ndynamism. But, still, *wow.*\n\n### Slow key wrapping\n\nIf you take a look at `tableGet()`, you'll see it's mostly a wrapper around a\ncall to `findEntry()` where the actual hash table lookup happens. To refresh\nyour memory, here it is in full:\n\n```c\nstatic Entry* findEntry(Entry* entries, int capacity,\n ObjString* key) {\n uint32_t index = key->hash % capacity;\n Entry* tombstone = NULL;\n\n for (;;) {\n Entry* entry = &entries[index];\n if (entry->key == NULL) {\n if (IS_NIL(entry->value)) {\n // Empty entry.\n return tombstone != NULL ? tombstone : entry;\n } else {\n // We found a tombstone.\n if (tombstone == NULL) tombstone = entry;\n }\n } else if (entry->key == key) {\n // We found the key.\n return entry;\n }\n\n index = (index + 1) % capacity;\n }\n}\n```\n\nWhen running that previous benchmark -- on my machine, at least -- the VM spends\n70% of the total execution time on *one line* in this function. Any guesses as\nto which one? No? It's this:\n\n```c\n uint32_t index = key->hash % capacity;\n```\n\nThat pointer dereference isn't the problem. It's the little `%`. It turns out\nthe modulo operator is *really* slow. Much slower than other arithmetic operators. Can we do something better?\n\n\n\nIn the general case, it's really hard to re-implement a fundamental arithmetic\noperator in user code in a way that's faster than what the CPU itself can do.\nAfter all, our C code ultimately compiles down to the CPU's own arithmetic\noperations. If there were tricks we could use to go faster, the chip would\nalready be using them.\n\nHowever, we can take advantage of the fact that we know more about our problem\nthan the CPU does. We use modulo here to take a key string's hash code and\nwrap it to fit within the bounds of the table's entry array. That array starts\nout at eight elements and grows by a factor of two each time. We know -- and the\nCPU and C compiler do not -- that our table's size is always a power of two.\n\nBecause we're clever bit twiddlers, we know a faster way to calculate the\nremainder of a number modulo a power of two: **bit masking**. Let's say we want\nto calculate 229 modulo 64. The answer is 37, which is not particularly apparent\nin decimal, but is clearer when you view those numbers in binary:\n\n\"The\n\nOn the left side of the illustration, notice how the result (37) is simply the\ndividend (229) with the highest two bits shaved off? Those two highest bits are\nthe bits at or to the left of the divisor's single 1 bit.\n\nOn the right side, we get the same result by taking 229 and bitwise AND-ing it with 63, which is one less than our\noriginal power of two divisor. Subtracting one from a power of two gives you a\nseries of 1 bits. That is exactly the mask we need in order to strip out those\ntwo leftmost bits.\n\nIn other words, you can calculate a number modulo any power of two by simply\nAND-ing it with that power of two minus one. I'm\nnot enough of a mathematician to *prove* to you that this works, but if you\nthink it through, it should make sense. We can replace that slow modulo operator\nwith a very fast decrement and bitwise AND. We\nsimply change the offending line of code to this:\n\n^code initial-index (2 before, 1 after)\n\nCPUs love bitwise operators, so it's hard to improve on that. \n\n\n\nOur linear probing search may need to wrap around the end of the array, so there\nis another modulo in `findEntry()` to update.\n\n^code next-index (4 before, 1 after)\n\nThis line didn't show up in the profiler since most searches don't wrap.\n\nThe `findEntry()` function has a sister function, `tableFindString()` that does\na hash table lookup for interning strings. We may as well apply the same\noptimizations there too. This function is called only when interning strings,\nwhich wasn't heavily stressed by our benchmark. But a Lox program that created\nlots of strings might noticeably benefit from this change.\n\n^code find-string-index (2 before, 2 after)\n\nAnd also when the linear probing wraps around.\n\n^code find-string-next (3 before, 1 after)\n\nLet's see if our fixes were worth it. I tweaked that zoological benchmark to\ncount how many batches of 10,000 calls it can run in\nten seconds. More batches equals faster performance. On my machine using the\nunoptimized code, the benchmark gets through 3,192 batches. After this\noptimization, that jumps to 6,249.\n\n\"Bar\n\nThat's almost exactly twice as much work in the same amount of time. We made the\nVM twice as fast (usual caveat: on this benchmark). That is a massive win when\nit comes to optimization. Usually you feel good if you can claw a few percentage\npoints here or there. Since methods, fields, and global variables are so\nprevalent in Lox programs, this tiny optimization improves performance across\nthe board. Almost every Lox program benefits.\n\n\n\nNow, the point of this section is *not* that the modulo operator is profoundly\nevil and you should stamp it out of every program you ever write. Nor is it that\nmicro-optimization is a vital engineering skill. It's rare that a performance\nproblem has such a narrow, effective solution. We got lucky.\n\nThe point is that we didn't *know* that the modulo operator was a performance\ndrain until our profiler told us so. If we had wandered around our VM's codebase\nblindly guessing at hotspots, we likely wouldn't have noticed it. What I want\nyou to take away from this is how important it is to have a profiler in your\ntoolbox.\n\nTo reinforce that point, let's go ahead and run the original benchmark in our\nnow-optimized VM and see what the profiler shows us. On my machine, `tableGet()`\nis still a fairly large chunk of execution time. That's to be expected for a\ndynamically typed language. But it has dropped from 72% of the total execution\ntime down to 35%. That's much more in line with what we'd like to see and shows\nthat our optimization didn't just make the program faster, but made it faster\n*in the way we expected*. Profilers are as useful for verifying solutions as\nthey are for discovering problems.\n\n## NaN Boxing\n\nThis next optimization has a very different feel. Thankfully, despite the odd\nname, it does not involve punching your grandmother. It's different, but not,\nlike, *that* different. With our previous optimization, the profiler told us\nwhere the problem was, and we merely had to use some ingenuity to come up with a\nsolution.\n\nThis optimization is more subtle, and its performance effects more scattered\nacross the virtual machine. The profiler won't help us come up with this.\nInstead, it was invented by someone thinking deeply\nabout the lowest levels of machine architecture.\n\n\n\nLike the heading says, this optimization is called **NaN boxing** or sometimes\n**NaN tagging**. Personally I like the latter name because \"boxing\" tends to imply\nsome kind of heap-allocated representation, but the former seems to be the more\nwidely used term. This technique changes how we represent values in the VM.\n\nOn a 64-bit machine, our Value type takes up 16 bytes. The struct has two\nfields, a type tag and a union for the payload. The largest fields in the union\nare an Obj pointer and a double, which are both 8 bytes. To keep the union field\naligned to an 8-byte boundary, the compiler adds padding after the tag too:\n\n\"Byte\n\nThat's pretty big. If we could cut that down, then the VM could pack more values\ninto the same amount of memory. Most computers have plenty of RAM these days, so\nthe direct memory savings aren't a huge deal. But a smaller representation means\nmore Values fit in a cache line. That means fewer cache misses, which affects\n*speed*.\n\nIf Values need to be aligned to their largest payload size, and a Lox number or\nObj pointer needs a full 8 bytes, how can we get any smaller? In a dynamically\ntyped language like Lox, each value needs to carry not just its payload, but\nenough additional information to determine the value's type at runtime. If a Lox\nnumber is already using the full 8 bytes, where could we squirrel away a couple\nof extra bits to tell the runtime \"this is a number\"?\n\nThis is one of the perennial problems for dynamic language hackers. It\nparticularly bugs them because statically typed languages don't generally have\nthis problem. The type of each value is known at compile time, so no extra\nmemory is needed at runtime to track it. When your C compiler compiles a 32-bit\nint, the resulting variable gets *exactly* 32 bits of storage.\n\nDynamic language folks hate losing ground to the static camp, so they've come up\nwith a number of very clever ways to pack type information and a payload into a\nsmall number of bits. NaN boxing is one of those. It's a particularly good fit\nfor languages like JavaScript and Lua, where all numbers are double-precision\nfloating point. Lox is in that same boat.\n\n### What is (and is not) a number?\n\nBefore we start optimizing, we need to really understand how our friend the CPU\nrepresents floating-point numbers. Almost all machines today use the same\nscheme, encoded in the venerable scroll [IEEE 754][754], known to mortals as the\n\"IEEE Standard for Floating-Point Arithmetic\".\n\n[754]: https://en.wikipedia.org/wiki/IEEE_754\n\nIn the eyes of your computer, a 64-bit,\ndouble-precision, IEEE floating-point number looks like this:\n\n\n\n\"Bit\n\n* Starting from the right, the first 52 bits are the **fraction**,\n **mantissa**, or **significand** bits. They represent the significant digits\n of the number, as a binary integer.\n\n* Next to that are 11 **exponent** bits. These tell you how far the mantissa\n is shifted away from the decimal (well, binary) point.\n\n* The highest bit is the **sign bit**, which\n indicates whether the number is positive or negative.\n\nI know that's a little vague, but this chapter isn't a deep dive on\nfloating point representation. If you want to know how the exponent and mantissa\nplay together, there are already better explanations out there than I could\nwrite.\n\n\n\nThe important part for our purposes is that the spec carves out a special case\nexponent. When all of the exponent bits are set, then instead of just\nrepresenting a really big number, the value has a different meaning. These\nvalues are \"Not a Number\" (hence, **NaN**) values. They represent concepts like\ninfinity or the result of division by zero.\n\n*Any* double whose exponent bits are all set is a NaN, regardless of the\nmantissa bits. That means there's lots and lots of *different* NaN bit patterns.\nIEEE 754 divides those into two categories. Values where the highest mantissa\nbit is 0 are called **signalling NaNs**, and the others are **quiet NaNs**.\nSignalling NaNs are intended to be the result of erroneous computations, like\ndivision by zero. A chip may detect when one of these\nvalues is produced and abort a program completely. They may self-destruct if you\ntry to read one.\n\n\n\nQuiet NaNs are supposed to be safer to use. They don't represent useful numeric\nvalues, but they should at least not set your hand on fire if you touch them.\n\nEvery double with all of its exponent bits set and its highest mantissa bit set\nis a quiet NaN. That leaves 52 bits unaccounted for. We'll avoid one of those so\nthat we don't step on Intel's \"QNaN Floating-Point Indefinite\" value, leaving us\n51 bits. Those remaining bits can be anything. We're talking\n2,251,799,813,685,248 unique quiet NaN bit patterns.\n\n\"The\n\nThis means a 64-bit double has enough room to store all of the various different\nnumeric floating-point values and *also* has room for another 51 bits of data\nthat we can use however we want. That's plenty of room to set aside a couple of\nbit patterns to represent Lox's `nil`, `true`, and `false` values. But what\nabout Obj pointers? Don't pointers need a full 64 bits too?\n\nFortunately, we have another trick up our other sleeve. Yes, technically\npointers on a 64-bit architecture are 64 bits. But, no architecture I know of\nactually uses that entire address space. Instead, most widely used chips today\nonly ever use the low 48 bits. The remaining 16 bits are\neither unspecified or always zero.\n\n\n\nIf we've got 51 bits, we can stuff a 48-bit pointer in there with three bits to\nspare. Those three bits are just enough to store tiny type tags to distinguish\nbetween `nil`, Booleans, and Obj pointers.\n\nThat's NaN boxing. Within a single 64-bit double, you can store all of the\ndifferent floating-point numeric values, a pointer, or any of a couple of other\nspecial sentinel values. Half the memory usage of our current Value struct,\nwhile retaining all of the fidelity.\n\nWhat's particularly nice about this representation is that there is no need to\n*convert* a numeric double value into a \"boxed\" form. Lox numbers *are* just\nnormal, 64-bit doubles. We still need to *check* their type before we use them,\nsince Lox is dynamically typed, but we don't need to do any bit shifting or\npointer indirection to go from \"value\" to \"number\".\n\nFor the other value types, there is a conversion step, of course. But,\nfortunately, our VM hides all of the mechanism to go from values to raw types\nbehind a handful of macros. Rewrite those to implement NaN boxing, and the rest\nof the VM should just work.\n\n### Conditional support\n\nI know the details of this new representation aren't clear in your head yet.\nDon't worry, they will crystallize as we work through the implementation. Before\nwe get to that, we're going to put some compile-time scaffolding in place.\n\nFor our previous optimization, we rewrote the previous slow code and called it\ndone. This one is a little different. NaN boxing relies on some very low-level\ndetails of how a chip represents floating-point numbers and pointers. It\n*probably* works on most CPUs you're likely to encounter, but you can never be\ntotally sure.\n\nIt would suck if our VM completely lost support for an architecture just because\nof its value representation. To avoid that, we'll maintain support for *both*\nthe old tagged union implementation of Value and the new NaN-boxed form. We\nselect which representation we want at compile time using this flag:\n\n^code define-nan-boxing (2 before, 1 after)\n\nIf that's defined, the VM uses the new form. Otherwise, it reverts to the old\nstyle. The few pieces of code that care about the details of the value\nrepresentation -- mainly the handful of macros for wrapping and unwrapping\nValues -- vary based on whether this flag is set. The rest of the VM can\ncontinue along its merry way.\n\nMost of the work happens in the \"value\" module where we add a section for the\nnew type.\n\n^code nan-boxing (2 before, 1 after)\n\nWhen NaN boxing is enabled, the actual type of a Value is a flat, unsigned\n64-bit integer. We could use double instead, which would make the macros for\ndealing with Lox numbers a little simpler. But all of the other macros need to\ndo bitwise operations and uint64_t is a much friendlier type for that. Outside\nof this module, the rest of the VM doesn't really care one way or the other.\n\nBefore we start re-implementing those macros, we close the `#else` branch of the\n`#ifdef` at the end of the definitions for the old representation.\n\n^code end-if-nan-boxing (1 before, 2 after)\n\nOur remaining task is simply to fill in that first `#ifdef` section with new\nimplementations of all the stuff already in the `#else` side. We'll work through\nit one value type at a time, from easiest to hardest.\n\n### Numbers\n\nWe'll start with numbers since they have the most direct representation under\nNaN boxing. To \"convert\" a C double to a NaN-boxed clox Value, we don't need to\ntouch a single bit -- the representation is exactly the same. But we do need to\nconvince our C compiler of that fact, which we made harder by defining Value to\nbe uint64_t.\n\nWe need to get the compiler to take a set of bits that it thinks are a double\nand use those same bits as a uint64_t, or vice versa. This is called **type\npunning**. C and C++ programmers have been doing this since the days of bell\nbottoms and 8-tracks, but the language specifications have hesitated to say which of the many ways to do this is\nofficially sanctioned.\n\n\n\nI know one way to convert a `double` to `Value` and back that I believe is\nsupported by both the C and C++ specs. Unfortunately, it doesn't fit in a single\nexpression, so the conversion macros have to call out to helper functions.\nHere's the first macro:\n\n^code number-val (1 before, 2 after)\n\nThat macro passes the double here:\n\n^code num-to-value (1 before, 2 after)\n\nI know, weird, right? The way to treat a series of bytes as having a different\ntype without changing their value at all is `memcpy()`? This looks horrendously\nslow: Create a local variable. Pass its address to the operating system through\na syscall to copy a few bytes. Then return the result, which is the exact same\nbytes as the input. Thankfully, because this *is* the supported idiom for type\npunning, most compilers recognize the pattern and optimize away the `memcpy()`\nentirely.\n\n\"Unwrapping\" a Lox number is the mirror image.\n\n^code as-number (1 before, 2 after)\n\nThat macro calls this function:\n\n^code value-to-num (1 before, 2 after)\n\nIt works exactly the same except we swap the types. Again, the compiler will\neliminate all of it. Even though those calls to\n`memcpy()` will disappear, we still need to show the compiler *which* `memcpy()`\nwe're calling so we also need an include.\n\n\n\n^code include-string (1 before, 2 after)\n\nThat was a lot of code to ultimately do nothing but silence the C type checker.\nDoing a runtime type *test* on a Lox number is a little more interesting. If all\nwe have are exactly the bits for a double, how do we tell that it *is* a double?\nIt's time to get bit twiddling.\n\n^code is-number (1 before, 2 after)\n\nWe know that every Value that is *not* a number will use a special quiet NaN\nrepresentation. And we presume we have correctly avoided any of the meaningful\nNaN representations that may actually be produced by doing arithmetic on\nnumbers.\n\nIf the double has all of its NaN bits set, and the quiet NaN bit set, and one\nmore for good measure, we can be pretty certain it\nis one of the bit patterns we ourselves have set aside for other types. To check\nthat, we mask out all of the bits except for our set of quiet NaN bits. If *all*\nof those bits are set, it must be a NaN-boxed value of some other Lox type.\nOtherwise, it is actually a number.\n\n\n\nThe set of quiet NaN bits are declared like this:\n\n^code qnan (1 before, 2 after)\n\nIt would be nice if C supported binary literals. But if you do the conversion,\nyou'll see that value is the same as this:\n\n\"The\n\nThis is exactly all of the exponent bits, plus the quiet NaN bit, plus one extra\nto dodge that Intel value.\n\n### Nil, true, and false\n\nThe next type to handle is `nil`. That's pretty simple since there's only one\n`nil` value and thus we need only a single bit pattern to represent it. There\nare two other singleton values, the two Booleans, `true` and `false`. This calls\nfor three total unique bit patterns.\n\nTwo bits give us four different combinations, which is plenty. We claim the two\nlowest bits of our unused mantissa space as a \"type tag\" to determine which of\nthese three singleton values we're looking at. The three type tags are defined\nlike so:\n\n^code tags (1 before, 2 after)\n\nOur representation of `nil` is thus all of the bits required to define our\nquiet NaN representation along with the `nil` type tag bits:\n\n\"The\n\nIn code, we check the bits like so:\n\n^code nil-val (2 before, 1 after)\n\nWe simply bitwise OR the quiet NaN bits and the\ntype tag, and then do a little cast dance to teach the C compiler what we want\nthose bits to mean.\n\nSince `nil` has only a single bit representation, we can use equality on\nuint64_t to see if a Value is `nil`.\n\n\n\n^code is-nil (2 before, 1 after)\n\nYou can guess how we define the `true` and `false` values.\n\n^code false-true-vals (2 before, 1 after)\n\nThe bits look like this:\n\n\"The\n\nTo convert a C bool into a Lox Boolean, we rely on these two singleton values\nand the good old conditional operator.\n\n^code bool-val (2 before, 1 after)\n\nThere's probably a cleverer bitwise way to do this, but my hunch is that the\ncompiler can figure one out faster than I can. Going the other direction is\nsimpler.\n\n^code as-bool (2 before, 1 after)\n\nSince we know there are exactly two Boolean bit representations in Lox -- unlike\nin C where any non-zero value can be considered \"true\" -- if it ain't `true`, it\nmust be `false`. This macro does assume you call it only on a Value that you\nknow *is* a Lox Boolean. To check that, there's one more macro.\n\n^code is-bool (2 before, 1 after)\n\nThat looks a little strange. A more obvious macro would look like this:\n\n```c\n#define IS_BOOL(v) ((v) == TRUE_VAL || (v) == FALSE_VAL)\n```\n\nUnfortunately, that's not safe. The expansion mentions `v` twice, which means if\nthat expression has any side effects, they will be executed twice. We could have\nthe macro call out to a separate function, but, ugh, what a chore.\n\nInstead, we bitwise OR a 1 onto the value to\nmerge the only two valid Boolean bit patterns. That leaves three potential\nstates the value can be in:\n\n1. It was `FALSE_VAL` and has now been converted to `TRUE_VAL`.\n\n2. It was `TRUE_VAL` and the `| 1` did nothing and it's still `TRUE_VAL`.\n\n3. It's some other, non-Boolean value.\n\nAt that point, we can simply compare the result to `TRUE_VAL` to see if we're\nin the first two states or the third.\n\n### Objects\n\nThe last value type is the hardest. Unlike the singleton values, there are\nbillions of different pointer values we need to box inside a NaN. This means we\nneed both some kind of tag to indicate that these particular NaNs *are* Obj\npointers, and room for the addresses themselves.\n\nThe tag bits we used for the singleton values are in the region where I decided\nto store the pointer itself, so we can't easily use a different bit there to indicate that the value is an object reference.\nHowever, there is another bit we aren't using. Since all our NaN values are not\nnumbers -- it's right there in the name -- the sign bit isn't used for anything.\nWe'll go ahead and use that as the type tag for objects. If one of our quiet\nNaNs has its sign bit set, then it's an Obj pointer. Otherwise, it must be one\nof the previous singleton values.\n\n\n\nIf the sign bit is set, then the remaining low bits store the pointer to the\nObj:\n\n\"Bit\n\nTo convert a raw Obj pointer to a Value, we take the pointer and set all of the\nquiet NaN bits and the sign bit.\n\n^code obj-val (1 before, 2 after)\n\nThe pointer itself is a full 64 bits, and in principle,\nit could thus overlap with some of those quiet NaN and sign bits. But in\npractice, at least on the architectures I've tested, everything above the 48th\nbit in a pointer is always zero. There's a lot of casting going on here, which\nI've found is necessary to satisfy some of the pickiest C compilers, but the\nend result is just jamming some bits together.\n\n\n\nWe define the sign bit like so:\n\n^code sign-bit (2 before, 2 after)\n\nTo get the Obj pointer back out, we simply mask off all of those extra bits.\n\n^code as-obj (1 before, 2 after)\n\nThe tilde (`~`), if you haven't done enough bit manipulation to encounter it\nbefore, is bitwise NOT. It toggles all ones and\nzeroes in its operand. By masking the value with the bitwise negation of the\nquiet NaN and sign bits, we *clear* those bits and let the pointer bits remain.\n\nOne last macro:\n\n^code is-obj (1 before, 2 after)\n\nA Value storing an Obj pointer has its sign bit set, but so does any negative\nnumber. To tell if a Value is an Obj pointer, we need to check that both the\nsign bit and all of the quiet NaN bits are set. This is similar to how we detect\nthe type of the singleton values, except this time we use the sign bit as the\ntag.\n\n### Value functions\n\nThe rest of the VM usually goes through the macros when working with Values, so\nwe are almost done. However, there are a couple of functions in the \"value\"\nmodule that peek inside the otherwise black box of Value and work with its\nencoding directly. We need to fix those too.\n\nThe first is `printValue()`. It has separate code for each value type. We no\nlonger have an explicit type enum we can switch on, so instead we use a series\nof type tests to handle each kind of value.\n\n^code print-value (1 before, 1 after)\n\nThis is technically a tiny bit slower than a switch, but compared to the\noverhead of actually writing to a stream, it's negligible.\n\nWe still support the original tagged union representation, so we keep the old\ncode and enclose it in the `#else` conditional section.\n\n^code end-print-value (1 before, 1 after)\n\nThe other operation is testing two values for equality.\n\n^code values-equal (1 before, 1 after)\n\nIt doesn't get much simpler than that! If the two bit representations are\nidentical, the values are equal. That does the right thing for the singleton\nvalues since each has a unique bit representation and they are only equal to\nthemselves. It also does the right thing for Obj pointers, since objects use\nidentity for equality -- two Obj references are equal only if they point to the\nexact same object.\n\nIt's *mostly* correct for numbers too. Most floating-point numbers with\ndifferent bit representations are distinct numeric values. Alas, IEEE 754\ncontains a pothole to trip us up. For reasons that aren't entirely clear to me,\nthe spec mandates that NaN values are *not* equal to *themselves*. This isn't a\nproblem for the special quiet NaNs that we are using for our own purposes. But\nit's possible to produce a \"real\" arithmetic NaN in Lox, and if we want to\ncorrectly implement IEEE 754 numbers, then the resulting value is not supposed\nto be equal to itself. More concretely:\n\n```lox\nvar nan = 0/0;\nprint nan == nan;\n```\n\nIEEE 754 says this program is supposed to print \"false\". It does the right thing\nwith our old tagged union representation because the `VAL_NUMBER` case applies\n`==` to two values that the C compiler knows are doubles. Thus the compiler\ngenerates the right CPU instruction to perform an IEEE floating-point equality.\n\nOur new representation breaks that by defining Value to be a uint64_t. If we\nwant to be *fully* compliant with IEEE 754, we need to handle this case.\n\n^code nan-equality (1 before, 1 after)\n\nI know, it's weird. And there is a performance cost to doing this type test\nevery time we check two Lox values for equality. If we are willing to sacrifice\na little compatibility -- who *really* cares if NaN is\nnot equal to itself? -- we could leave this off. I'll leave it up to you to\ndecide how pedantic you want to be.\n\n\n\nFinally, we close the conditional compilation section around the old\nimplementation.\n\n^code end-values-equal (1 before, 1 after)\n\nAnd that's it. This optimization is complete, as is our clox virtual machine.\nThat was the last line of new code in the book.\n\n### Evaluating performance\n\nThe code is done, but we still need to figure out if we actually made anything\nbetter with these changes. Evaluating an optimization like this is very\ndifferent from the previous one. There, we had a clear hotspot visible in the\nprofiler. We fixed that part of the code and could instantly see the hotspot\nget faster.\n\nThe effects of changing the value representation are more diffused. The macros\nare expanded in place wherever they are used, so the performance changes are\nspread across the codebase in a way that's hard for many profilers to track\nwell, especially in an optimized build.\n\n\n\nWe also can't easily *reason* about the effects of our change. We've made values\nsmaller, which reduces cache misses all across the VM. But the actual real-world\nperformance effect of that change is highly dependent on the memory use of the\nLox program being run. A tiny Lox microbenchmark may not have enough values\nscattered around in memory for the effect to be noticeable, and even things like\nthe addresses handed out to us by the C memory allocator can impact the results.\n\nIf we did our job right, basically everything gets a little faster, especially\non larger, more complex Lox programs. But it is possible that the extra bitwise\noperations we do when NaN-boxing values nullify the gains from the better\nmemory use. Doing performance work like this is unnerving because you can't\neasily *prove* that you've made the VM better. You can't point to a single\nsurgically targeted microbenchmark and say, \"There, see?\"\n\nInstead, what we really need is a *suite* of larger benchmarks. Ideally, they\nwould be distilled from real-world applications -- not that such a thing exists\nfor a toy language like Lox. Then we can measure the aggregate performance\nchanges across all of those. I did my best to cobble together a handful of\nlarger Lox programs. On my machine, the new value representation seems to make\neverything roughly 10% faster across the board.\n\nThat's not a huge improvement, especially compared to the profound effect of\nmaking hash table lookups faster. I added this optimization in large part\nbecause it's a good example of a certain *kind* of performance work you may\nexperience, and honestly, because I think it's technically really cool. It might\nnot be the first thing I would reach for if I were seriously trying to make clox\nfaster. There is probably other, lower-hanging fruit.\n\nBut, if you find yourself working on a program where all of the easy wins have\nbeen taken, then at some point you may want to think about tuning your value\nrepresentation. I hope this chapter has shined a light on some of the options\nyou have in that area.\n\n## Where to Next\n\nWe'll stop here with the Lox language and our two interpreters. We could tinker\non it forever, adding new language features and clever speed improvements. But,\nfor this book, I think we've reached a natural place to call our work complete.\nI won't rehash everything we've learned in the past many pages. You were there\nwith me and you remember. Instead, I'd like to take a minute to talk about where\nyou might go from here. What is the next step in your programming language\njourney?\n\nMost of you probably won't spend a significant part of your career working in\ncompilers or interpreters. It's a pretty small slice of the computer science\nacademia pie, and an even smaller segment of software engineering in industry.\nThat's OK. Even if you never work on a compiler again in your life, you will\ncertainly *use* one, and I hope this book has equipped you with a better\nunderstanding of how the programming languages you use are designed and\nimplemented.\n\nYou have also learned a handful of important, fundamental data structures and\ngotten some practice doing low-level profiling and optimization work. That kind\nof expertise is helpful no matter what domain you program in.\n\nI also hope I gave you a new way of looking at and\nsolving problems. Even if you never work on a language again, you may be\nsurprised to discover how many programming problems can be seen as\nlanguage-*like*. Maybe that report generator you need to write can be modeled as\na series of stack-based \"instructions\" that the generator \"executes\". That user\ninterface you need to render looks an awful lot like traversing an AST.\n\n\n\nIf you do want to go further down the programming language rabbit hole, here\nare some suggestions for which branches in the tunnel to explore:\n\n* Our simple, single-pass bytecode compiler pushed us towards mostly runtime\n optimization. In a mature language implementation, compile-time optimization\n is generally more important, and the field of compiler optimizations is\n incredibly rich. Grab a classic compilers book,\n and rebuild the front end of clox or jlox to be a sophisticated compilation\n pipeline with some interesting intermediate representations and optimization\n passes.\n\n Dynamic typing will place some restrictions on how far you can go, but there\n is still a lot you can do. Or maybe you want to take a big leap and add\n static types and a type checker to Lox. That will certainly give your front\n end a lot more to chew on.\n\n \n\n* In this book, I aim to be correct, but not particularly rigorous. My goal is\n mostly to give you an *intuition* and a feel for doing language work. If you\n like more precision, then the whole world of programming language academia\n is waiting for you. Languages and compilers have been studied formally since\n before we even had computers, so there is no shortage of books and papers on\n parser theory, type systems, semantics, and formal logic. Going down this\n path will also teach you how to read CS papers, which is a valuable skill in\n its own right.\n\n* Or, if you just really enjoy hacking on and making languages, you can take\n Lox and turn it into your own plaything. Change\n the syntax to something that delights your eye. Add missing features or\n remove ones you don't like. Jam new optimizations in there.\n\n \n\n Eventually you may get to a point where you have something you think others\n could use as well. That gets you into the very distinct world of programming\n language *popularity*. Expect to spend a ton of time writing documentation,\n example programs, tools, and useful libraries. The field is crowded with\n languages vying for users. To thrive in that space you'll have to put on\n your marketing hat and *sell*. Not everyone enjoys that kind of\n public-facing work, but if you do, it can be incredibly gratifying to see\n people use your language to express themselves.\n\nOr maybe this book has satisfied your craving and you'll stop here. Whichever\nway you go, or don't go, there is one lesson I hope to lodge in your heart. Like\nI was, you may have initially been intimidated by programming languages. But in\nthese chapters, you've seen that even really challenging material can be tackled\nby us mortals if we get our hands dirty and take it a step at a time. If you can\nhandle compilers and interpreters, you can do anything you put your mind to.\n\n[mit license]: https://en.wikipedia.org/wiki/MIT_License\n[source]: https://github.com/munificent/craftinginterpreters\n\n
\n\n## Challenges\n\nAssigning homework on the last day of school seems cruel but if you really want\nsomething to do during your summer vacation:\n\n1. Fire up your profiler, run a couple of benchmarks, and look for other\n hotspots in the VM. Do you see anything in the runtime that you can improve?\n\n2. Many strings in real-world user programs are small, often only a character\n or two. This is less of a concern in clox because we intern strings, but\n most VMs don't. For those that don't, heap allocating a tiny character array\n for each of those little strings and then representing the value as a\n pointer to that array is wasteful. Often, the pointer is larger than the\n string's characters. A classic trick is to have a separate value\n representation for small strings that stores the characters inline in the\n value.\n\n Starting from clox's original tagged union representation, implement that\n optimization. Write a couple of relevant benchmarks and see if it helps.\n\n3. Reflect back on your experience with this book. What parts of it worked well\n for you? What didn't? Was it easier for you to learn bottom-up or top-down?\n Did the illustrations help or distract? Did the analogies clarify or\n confuse?\n\n The more you understand your personal learning style, the more effectively\n you can upload knowledge into your head. You can specifically target\n material that teaches you the way you learn best.\n\n
\n" }, { "path": "book/parsing-expressions.md", "content": "> Grammar, which knows how to control even kings.\n> Molière\n\nThis chapter marks the first major milestone of the\nbook. Many of us have cobbled together a mishmash of regular expressions and\nsubstring operations to extract some sense out of a pile of text. The code was\nprobably riddled with bugs and a beast to maintain. Writing a *real* parser --\none with decent error handling, a coherent internal structure, and the ability\nto robustly chew through a sophisticated syntax -- is considered a rare,\nimpressive skill. In this chapter, you will attain\nit.\n\n\n\n\n\nIt's easier than you think, partially because we front-loaded a lot of the hard\nwork in the [last chapter][]. You already know your way around a formal grammar.\nYou're familiar with syntax trees, and we have some Java classes to represent\nthem. The only remaining piece is parsing -- transmogrifying a sequence of\ntokens into one of those syntax trees.\n\n[last chapter]: representing-code.html\n\nSome CS textbooks make a big deal out of parsers. In the '60s, computer\nscientists -- understandably tired of programming in assembly language --\nstarted designing more sophisticated, human-friendly\nlanguages like Fortran and ALGOL. Alas, they weren't very *machine*-friendly\nfor the primitive computers of the time.\n\n\n\nThese pioneers designed languages that they honestly weren't even sure how to\nwrite compilers for, and then did groundbreaking work inventing parsing and\ncompiling techniques that could handle these new, big languages on those old, tiny\nmachines.\n\nClassic compiler books read like fawning hagiographies of these heroes and their\ntools. The cover of *Compilers: Principles, Techniques, and Tools* literally has\na dragon labeled \"complexity of compiler design\" being slain by a knight bearing\na sword and shield branded \"LALR parser generator\" and \"syntax directed\ntranslation\". They laid it on thick.\n\nA little self-congratulation is well-deserved, but the truth is you don't need\nto know most of that stuff to bang out a high quality parser for a modern\nmachine. As always, I encourage you to broaden your education and take it in\nlater, but this book omits the trophy case.\n\n## Ambiguity and the Parsing Game\n\nIn the last chapter, I said you can \"play\" a context-free grammar like a game in\norder to *generate* strings. Parsers play that game in reverse. Given a string\n-- a series of tokens -- we map those tokens to terminals in the grammar to\nfigure out which rules could have generated that string.\n\nThe \"could have\" part is interesting. It's entirely possible to create a grammar\nthat is *ambiguous*, where different choices of productions can lead to the same\nstring. When you're using the grammar to *generate* strings, that doesn't matter\nmuch. Once you have the string, who cares how you got to it?\n\nWhen parsing, ambiguity means the parser may misunderstand the user's code. As\nwe parse, we aren't just determining if the string is valid Lox code, we're\nalso tracking which rules match which parts of it so that we know what part of\nthe language each token belongs to. Here's the Lox expression grammar we put\ntogether in the last chapter:\n\n```ebnf\nexpression → literal\n | unary\n | binary\n | grouping ;\n\nliteral → NUMBER | STRING | \"true\" | \"false\" | \"nil\" ;\ngrouping → \"(\" expression \")\" ;\nunary → ( \"-\" | \"!\" ) expression ;\nbinary → expression operator expression ;\noperator → \"==\" | \"!=\" | \"<\" | \"<=\" | \">\" | \">=\"\n | \"+\" | \"-\" | \"*\" | \"/\" ;\n```\n\nThis is a valid string in that grammar:\n\n\"6\n\nBut there are two ways we could have generated it. One way is:\n\n1. Starting at `expression`, pick `binary`.\n2. For the left-hand `expression`, pick `NUMBER`, and use `6`.\n3. For the operator, pick `\"/\"`.\n4. For the right-hand `expression`, pick `binary` again.\n5. In that nested `binary` expression, pick `3 - 1`.\n\nAnother is:\n\n1. Starting at `expression`, pick `binary`.\n2. For the left-hand `expression`, pick `binary` again.\n3. In that nested `binary` expression, pick `6 / 3`.\n4. Back at the outer `binary`, for the operator, pick `\"-\"`.\n5. For the right-hand `expression`, pick `NUMBER`, and use `1`.\n\nThose produce the same *strings*, but not the same *syntax trees*:\n\n\"Two\n\nIn other words, the grammar allows seeing the expression as `(6 / 3) - 1` or `6\n/ (3 - 1)`. The `binary` rule lets operands nest any which way you want. That in\nturn affects the result of evaluating the parsed tree. The way mathematicians\nhave addressed this ambiguity since blackboards were first invented is by\ndefining rules for precedence and associativity.\n\n* **Precedence** determines which operator\n is evaluated first in an expression containing a mixture of different\n operators. Precedence rules tell us that we evaluate the `/` before the `-`\n in the above example. Operators with higher precedence are evaluated\n before operators with lower precedence. Equivalently, higher precedence\n operators are said to \"bind tighter\".\n\n* **Associativity** determines which operator is evaluated first in a series\n of the *same* operator. When an operator is **left-associative** (think\n \"left-to-right\"), operators on the left evaluate before those on the right.\n Since `-` is left-associative, this expression:\n\n ```lox\n 5 - 3 - 1\n ```\n\n is equivalent to:\n\n ```lox\n (5 - 3) - 1\n ```\n\n Assignment, on the other hand, is **right-associative**. This:\n\n ```lox\n a = b = c\n ```\n\n is equivalent to:\n\n ```lox\n a = (b = c)\n ```\n\n\n\nWithout well-defined precedence and associativity, an expression that uses\nmultiple operators is ambiguous -- it can be parsed into different syntax trees,\nwhich could in turn evaluate to different results. We'll fix that in Lox by\napplying the same precedence rules as C, going from lowest to highest.\n\n\n\n\n \n \n \n\n\n\n\n \n \n \n\n\n \n \n \n\n\n \n \n \n\n\n \n \n \n\n\n \n \n \n\n\n
NameOperatorsAssociates
Equality== !=Left
Comparison> >=\n < <=Left
Term- +Left
Factor/ *Left
Unary! -Right
\n\nRight now, the grammar stuffs all expression types into a single `expression`\nrule. That same rule is used as the non-terminal for operands, which lets the\ngrammar accept any kind of expression as a subexpression, regardless of whether\nthe precedence rules allow it.\n\nWe fix that by stratifying the grammar. We define a\nseparate rule for each precedence level.\n\n```ebnf\nexpression → ...\nequality → ...\ncomparison → ...\nterm → ...\nfactor → ...\nunary → ...\nprimary → ...\n```\n\n\n\nEach rule here only matches expressions at its precedence level or higher. For\nexample, `unary` matches a unary expression like `!negated` or a primary\nexpression like `1234`. And `term` can match `1 + 2` but also `3 * 4 / 5`. The\nfinal `primary` rule covers the highest-precedence forms -- literals and\nparenthesized expressions.\n\nWe just need to fill in the productions for each of those rules. We'll do the\neasy ones first. The top `expression` rule matches any expression at any\nprecedence level. Since `equality` has the lowest\nprecedence, if we match that, then it covers everything.\n\n\n\n```ebnf\nexpression → equality\n```\n\nOver at the other end of the precedence table, a primary expression contains\nall the literals and grouping expressions.\n\n```ebnf\nprimary → NUMBER | STRING | \"true\" | \"false\" | \"nil\"\n | \"(\" expression \")\" ;\n```\n\nA unary expression starts with a unary operator followed by the operand. Since\nunary operators can nest -- `!!true` is a valid if weird expression -- the\noperand can itself be a unary operator. A recursive rule handles that nicely.\n\n```ebnf\nunary → ( \"!\" | \"-\" ) unary ;\n```\n\nBut this rule has a problem. It never terminates.\n\nRemember, each rule needs to match expressions at that precedence level *or\nhigher*, so we also need to let this match a primary expression.\n\n```ebnf\nunary → ( \"!\" | \"-\" ) unary\n | primary ;\n```\n\nThat works.\n\nThe remaining rules are all binary operators. We'll start with the rule for\nmultiplication and division. Here's a first try:\n\n```ebnf\nfactor → factor ( \"/\" | \"*\" ) unary\n | unary ;\n```\n\nThe rule recurses to match the left operand. That enables the rule to match a\nseries of multiplication and division expressions like `1 * 2 / 3`. Putting the\nrecursive production on the left side and `unary` on the right makes the rule\nleft-associative and unambiguous.\n\n\n\nAll of this is correct, but the fact that the first symbol in the body of the\nrule is the same as the head of the rule means this production is\n**left-recursive**. Some parsing techniques, including the one we're going to\nuse, have trouble with left recursion. (Recursion elsewhere, like we have in\n`unary` and the indirect recursion for grouping in `primary` are not a problem.)\n\nThere are many grammars you can define that match the same language. The choice\nfor how to model a particular language is partially a matter of taste and\npartially a pragmatic one. This rule is correct, but not optimal for how we\nintend to parse it. Instead of a left recursive rule, we'll use a different one.\n\n```ebnf\nfactor → unary ( ( \"/\" | \"*\" ) unary )* ;\n```\n\nWe define a factor expression as a flat *sequence* of multiplications\nand divisions. This matches the same syntax as the previous rule, but better\nmirrors the code we'll write to parse Lox. We use the same structure for all of\nthe other binary operator precedence levels, giving us this complete expression\ngrammar:\n\n```ebnf\nexpression → equality ;\nequality → comparison ( ( \"!=\" | \"==\" ) comparison )* ;\ncomparison → term ( ( \">\" | \">=\" | \"<\" | \"<=\" ) term )* ;\nterm → factor ( ( \"-\" | \"+\" ) factor )* ;\nfactor → unary ( ( \"/\" | \"*\" ) unary )* ;\nunary → ( \"!\" | \"-\" ) unary\n | primary ;\nprimary → NUMBER | STRING | \"true\" | \"false\" | \"nil\"\n | \"(\" expression \")\" ;\n```\n\nThis grammar is more complex than the one we had before, but in return we have\neliminated the previous one's ambiguity. It's just what we need to make a\nparser.\n\n## Recursive Descent Parsing\n\nThere is a whole pack of parsing techniques whose names are mostly combinations\nof \"L\" and \"R\" -- [LL(k)][], [LR(1)][lr], [LALR][] -- along with more exotic\nbeasts like [parser combinators][], [Earley parsers][], [the shunting yard\nalgorithm][yard], and [packrat parsing][]. For our first interpreter, one\ntechnique is more than sufficient: **recursive descent**.\n\n[ll(k)]: https://en.wikipedia.org/wiki/LL_parser\n[lr]: https://en.wikipedia.org/wiki/LR_parser\n[lalr]: https://en.wikipedia.org/wiki/LALR_parser\n[parser combinators]: https://en.wikipedia.org/wiki/Parser_combinator\n[earley parsers]: https://en.wikipedia.org/wiki/Earley_parser\n[yard]: https://en.wikipedia.org/wiki/Shunting-yard_algorithm\n[packrat parsing]: https://en.wikipedia.org/wiki/Parsing_expression_grammar\n\nRecursive descent is the simplest way to build a parser, and doesn't require\nusing complex parser generator tools like Yacc, Bison or ANTLR. All you need is\nstraightforward handwritten code. Don't be fooled by its simplicity, though.\nRecursive descent parsers are fast, robust, and can support sophisticated\nerror handling. In fact, GCC, V8 (the JavaScript VM in Chrome), Roslyn (the C#\ncompiler written in C#) and many other heavyweight production language\nimplementations use recursive descent. It rocks.\n\nRecursive descent is considered a **top-down parser** because it starts from the\ntop or outermost grammar rule (here `expression`) and works its way down into the nested subexpressions before finally\nreaching the leaves of the syntax tree. This is in contrast with bottom-up\nparsers like LR that start with primary expressions and compose them into larger\nand larger chunks of syntax.\n\n\n\nA recursive descent parser is a literal translation of the grammar's rules\nstraight into imperative code. Each rule becomes a function. The body of the\nrule translates to code roughly like:\n\n\n\n\n \n \n\n\n\n \n \n \n \n \n\n
Grammar notationCode representation
TerminalCode to match and consume a token
NonterminalCall to that rule’s function
|if or switch statement
* or +while or for loop
?if statement
\n\nThe descent is described as \"recursive\" because when a grammar rule refers to\nitself -- directly or indirectly -- that translates to a recursive function\ncall.\n\n### The parser class\n\nEach grammar rule becomes a method inside this new class:\n\n^code parser\n\nLike the scanner, the parser consumes a flat input sequence, only now we're\nreading tokens instead of characters. We store the list of tokens and use\n`current` to point to the next token eagerly waiting to be parsed.\n\nWe're going to run straight through the expression grammar now and translate\neach rule to Java code. The first rule, `expression`, simply expands to the\n`equality` rule, so that's straightforward.\n\n^code expression\n\nEach method for parsing a grammar rule produces a syntax tree for that rule and\nreturns it to the caller. When the body of the rule contains a nonterminal -- a\nreference to another rule -- we call that other rule's\nmethod.\n\n\n\nThe rule for equality is a little more complex.\n\n```ebnf\nequality → comparison ( ( \"!=\" | \"==\" ) comparison )* ;\n```\n\nIn Java, that becomes:\n\n^code equality\n\nLet's step through it. The first `comparison` nonterminal in the body translates\nto the first call to `comparison()` in the method. We take that result and store\nit in a local variable.\n\nThen, the `( ... )*` loop in the rule maps to a `while` loop. We need to know\nwhen to exit that loop. We can see that inside the rule, we must first find\neither a `!=` or `==` token. So, if we *don't* see one of those, we must be done\nwith the sequence of equality operators. We express that check using a handy\n`match()` method.\n\n^code match\n\nThis checks to see if the current token has any of the given types. If so, it\nconsumes the token and returns `true`. Otherwise, it returns `false` and leaves\nthe current token alone. The `match()` method is defined in terms of two more\nfundamental operations.\n\nThe `check()` method returns `true` if the current token is of the given type.\nUnlike `match()`, it never consumes the token, it only looks at it.\n\n^code check\n\nThe `advance()` method consumes the current token and returns it, similar to how\nour scanner's corresponding method crawled through characters.\n\n^code advance\n\nThese methods bottom out on the last handful of primitive operations.\n\n^code utils\n\n`isAtEnd()` checks if we've run out of tokens to parse. `peek()` returns the\ncurrent token we have yet to consume, and `previous()` returns the most recently\nconsumed token. The latter makes it easier to use `match()` and then access the\njust-matched token.\n\nThat's most of the parsing infrastructure we need. Where were we? Right, so if\nwe are inside the `while` loop in `equality()`, then we know we have found a\n`!=` or `==` operator and must be parsing an equality expression.\n\nWe grab the matched operator token so we can track which kind of equality\nexpression we have. Then we call `comparison()` again to parse the right-hand\noperand. We combine the operator and its two operands into a new `Expr.Binary`\nsyntax tree node, and then loop around. For each iteration, we store the\nresulting expression back in the same `expr` local variable. As we zip through a\nsequence of equality expressions, that creates a left-associative nested tree of\nbinary operator nodes.\n\n\n\n\"The\n\n\n\nThe parser falls out of the loop once it hits a token that's not an equality\noperator. Finally, it returns the expression. Note that if the parser never\nencounters an equality operator, then it never enters the loop. In that case,\nthe `equality()` method effectively calls and returns `comparison()`. In that\nway, this method matches an equality operator *or anything of higher\nprecedence*.\n\nMoving on to the next rule...\n\n```ebnf\ncomparison → term ( ( \">\" | \">=\" | \"<\" | \"<=\" ) term )* ;\n```\n\nTranslated to Java:\n\n^code comparison\n\nThe grammar rule is virtually identical to `equality`\nand so is the corresponding code. The only differences are the token types for\nthe operators we match, and the method we call for the operands -- now\n`term()` instead of `comparison()`. The remaining two binary operator rules\nfollow the same pattern.\n\nIn order of precedence, first addition and subtraction:\n\n\n\n^code term\n\nAnd finally, multiplication and division:\n\n^code factor\n\nThat's all of the binary operators, parsed with the correct precedence and\nassociativity. We're crawling up the precedence hierarchy and now we've reached\nthe unary operators.\n\n```ebnf\nunary → ( \"!\" | \"-\" ) unary\n | primary ;\n```\n\nThe code for this is a little different.\n\n^code unary\n\nAgain, we look at the current token to see how to\nparse. If it's a `!` or `-`, we must have a unary expression. In that case, we\ngrab the token and then recursively call `unary()` again to parse the operand.\nWrap that all up in a unary expression syntax tree and we're done.\n\n\n\nOtherwise, we must have reached the highest level of precedence, primary\nexpressions.\n\n```ebnf\nprimary → NUMBER | STRING | \"true\" | \"false\" | \"nil\"\n | \"(\" expression \")\" ;\n```\n\nMost of the cases for the rule are single terminals, so parsing is\nstraightforward.\n\n^code primary\n\nThe interesting branch is the one for handling parentheses. After we match an\nopening `(` and parse the expression inside it, we *must* find a `)` token. If\nwe don't, that's an error.\n\n## Syntax Errors\n\nA parser really has two jobs:\n\n1. Given a valid sequence of tokens, produce a corresponding syntax tree.\n\n2. Given an *invalid* sequence of tokens, detect any errors and tell the\n user about their mistakes.\n\nDon't underestimate how important the second job is! In modern IDEs and editors,\nthe parser is constantly reparsing code -- often while the user is still editing\nit -- in order to syntax highlight and support things like auto-complete. That\nmeans it will encounter code in incomplete, half-wrong states *all the time.*\n\nWhen the user doesn't realize the syntax is wrong, it is up to the parser to\nhelp guide them back onto the right path. The way it reports errors is a large\npart of your language's user interface. Good syntax error handling is hard. By\ndefinition, the code isn't in a well-defined state, so there's no infallible way\nto know what the user *meant* to write. The parser can't read your mind.\n\n\n\nThere are a couple of hard requirements for when the parser runs into a syntax\nerror. A parser must:\n\n* **Detect and report the error.** If it doesn't detect the error and passes the resulting malformed syntax tree on\n to the interpreter, all manner of horrors may be summoned.\n\n \n\n* **Avoid crashing or hanging.** Syntax errors are a fact of life, and\n language tools have to be robust in the face of them. Segfaulting or getting\n stuck in an infinite loop isn't allowed. While the source may not be valid\n *code*, it's still a valid *input to the parser* because users use the\n parser to learn what syntax is allowed.\n\nThose are the table stakes if you want to get in the parser game at all, but you\nreally want to raise the ante beyond that. A decent parser should:\n\n* **Be fast.** Computers are thousands of times faster than they were when\n parser technology was first invented. The days of needing to optimize your\n parser so that it could get through an entire source file during a coffee\n break are over. But programmer expectations have risen as quickly, if not\n faster. They expect their editors to reparse files in milliseconds after\n every keystroke.\n\n* **Report as many distinct errors as there are.** Aborting after the first\n error is easy to implement, but it's annoying for users if every time they\n fix what they think is the one error in a file, a new one appears. They\n want to see them all.\n\n* **Minimize *cascaded* errors.** Once a single error is found, the parser no\n longer really knows what's going on. It tries to get itself back on track\n and keep going, but if it gets confused, it may report a slew of ghost\n errors that don't indicate other real problems in the code. When the first\n error is fixed, those phantoms disappear, because they reflect only the\n parser's own confusion. Cascaded errors are annoying because they can scare\n the user into thinking their code is in a worse state than it is.\n\nThe last two points are in tension. We want to report as many separate errors as\nwe can, but we don't want to report ones that are merely side effects of an\nearlier one.\n\nThe way a parser responds to an error and keeps going to look for later errors\nis called **error recovery**. This was a hot research topic in the '60s. Back\nthen, you'd hand a stack of punch cards to the secretary and come back the next\nday to see if the compiler succeeded. With an iteration loop that slow, you\n*really* wanted to find every single error in your code in one pass.\n\nToday, when parsers complete before you've even finished typing, it's less of an\nissue. Simple, fast error recovery is fine.\n\n### Panic mode error recovery\n\n\n\nOf all the recovery techniques devised in yesteryear, the one that best stood\nthe test of time is called -- somewhat alarmingly -- **panic\nmode**. As soon as the parser detects an error, it enters panic mode. It\nknows at least one token doesn't make sense given its current state in the\nmiddle of some stack of grammar productions.\n\nBefore it can get back to parsing, it needs to get its state and the sequence of\nforthcoming tokens aligned such that the next token does match the rule being\nparsed. This process is called **synchronization**.\n\nTo do that, we select some rule in the grammar that will mark the\nsynchronization point. The parser fixes its parsing state by jumping out of any\nnested productions until it gets back to that rule. Then it synchronizes the\ntoken stream by discarding tokens until it reaches one that can appear at that\npoint in the rule.\n\nAny additional real syntax errors hiding in those discarded tokens aren't\nreported, but it also means that any mistaken cascaded errors that are side\neffects of the initial error aren't *falsely* reported either, which is a decent\ntrade-off.\n\nThe traditional place in the grammar to synchronize is between statements. We\ndon't have those yet, so we won't actually synchronize in this chapter, but\nwe'll get the machinery in place for later.\n\n### Entering panic mode\n\nBack before we went on this side trip around error recovery, we were writing the\ncode to parse a parenthesized expression. After parsing the expression, the\nparser looks for the closing `)` by calling `consume()`. Here, finally, is that\nmethod:\n\n^code consume\n\nIt's similar to `match()` in that it checks to see if the next token is of the\nexpected type. If so, it consumes the token and everything is groovy. If some\nother token is there, then we've hit an error. We report it by calling this:\n\n^code error\n\nFirst, that shows the error to the user by calling:\n\n^code token-error\n\nThis reports an error at a given token. It shows the token's location and the\ntoken itself. This will come in handy later since we use tokens throughout the\ninterpreter to track locations in code.\n\nAfter we report the error, the user knows about their mistake, but what does the\n*parser* do next? Back in `error()`, we create and return a ParseError, an\ninstance of this new class:\n\n^code parse-error (1 before, 1 after)\n\nThis is a simple sentinel class we use to unwind the parser. The `error()`\nmethod *returns* the error instead of *throwing* it because we want to let the\ncalling method inside the parser decide whether to unwind or not. Some parse\nerrors occur in places where the parser isn't likely to get into a weird state\nand we don't need to synchronize. In those\nplaces, we simply report the error and keep on truckin'.\n\nFor example, Lox limits the number of arguments you can pass to a function. If\nyou pass too many, the parser needs to report that error, but it can and should\nsimply keep on parsing the extra arguments instead of freaking out and going\ninto panic mode.\n\n\n\nIn our case, though, the syntax error is nasty enough that we want to panic and\nsynchronize. Discarding tokens is pretty easy, but how do we synchronize the\nparser's own state?\n\n### Synchronizing a recursive descent parser\n\nWith recursive descent, the parser's state -- which rules it is in the middle of\nrecognizing -- is not stored explicitly in fields. Instead, we use Java's\nown call stack to track what the parser is doing. Each rule in the middle of\nbeing parsed is a call frame on the stack. In order to reset that state, we need\nto clear out those call frames.\n\nThe natural way to do that in Java is exceptions. When we want to synchronize,\nwe *throw* that ParseError object. Higher up in the method for the grammar rule\nwe are synchronizing to, we'll catch it. Since we synchronize on statement\nboundaries, we'll catch the exception there. After the exception is caught, the\nparser is in the right state. All that's left is to synchronize the tokens.\n\nWe want to discard tokens until we're right at the beginning of the next\nstatement. That boundary is pretty easy to spot -- it's one of the main reasons\nwe picked it. *After* a semicolon, we're probably\nfinished with a statement. Most statements start with a keyword -- `for`, `if`,\n`return`, `var`, etc. When the *next* token is any of those, we're probably\nabout to start a statement.\n\n\n\nThis method encapsulates that logic:\n\n^code synchronize\n\nIt discards tokens until it thinks it has found a statement boundary. After\ncatching a ParseError, we'll call this and then we are hopefully back in sync.\nWhen it works well, we have discarded tokens that would have likely caused\ncascaded errors anyway, and now we can parse the rest of the file starting at\nthe next statement.\n\nAlas, we don't get to see this method in action, since we don't have statements\nyet. We'll get to that [in a couple of chapters][statements]. For now, if an\nerror occurs, we'll panic and unwind all the way to the top and stop parsing.\nSince we can parse only a single expression anyway, that's no big loss.\n\n[statements]: statements-and-state.html\n\n## Wiring up the Parser\n\nWe are mostly done parsing expressions now. There is one other place where we\nneed to add a little error handling. As the parser descends through the parsing\nmethods for each grammar rule, it eventually hits `primary()`. If none of the\ncases in there match, it means we are sitting on a token that can't start an\nexpression. We need to handle that error too.\n\n^code primary-error (5 before, 1 after)\n\nWith that, all that remains in the parser is to define an initial method to kick\nit off. That method is called, naturally enough, `parse()`.\n\n^code parse\n\nWe'll revisit this method later when we add statements to the language. For now,\nit parses a single expression and returns it. We also have some temporary code\nto exit out of panic mode. Syntax error recovery is the parser's job, so we\ndon't want the ParseError exception to escape into the rest of the interpreter.\n\nWhen a syntax error does occur, this method returns `null`. That's OK. The\nparser promises not to crash or hang on invalid syntax, but it doesn't promise\nto return a *usable syntax tree* if an error is found. As soon as the parser\nreports an error, `hadError` gets set, and subsequent phases are skipped.\n\nFinally, we can hook up our brand new parser to the main Lox class and try it\nout. We still don't have an interpreter, so for now, we'll parse to a syntax\ntree and then use the AstPrinter class from the [last chapter][ast-printer] to\ndisplay it.\n\n[ast-printer]: representing-code.html#a-not-very-pretty-printer\n\nDelete the old code to print the scanned tokens and replace it with this:\n\n^code print-ast (1 before, 1 after)\n\nCongratulations, you have crossed the threshold! That\nreally is all there is to handwriting a parser. We'll extend the grammar in\nlater chapters with assignment, statements, and other stuff, but none of that is\nany more complex than the binary operators we tackled here.\n\n\n\nFire up the interpreter and type in some expressions. See how it handles\nprecedence and associativity correctly? Not bad for less than 200 lines of code.\n\n
\n\n## Challenges\n\n1. In C, a block is a statement form that allows you to pack a series of\n statements where a single one is expected. The [comma operator][] is an\n analogous syntax for expressions. A comma-separated series of expressions\n can be given where a single expression is expected (except inside a function\n call's argument list). At runtime, the comma operator evaluates the left\n operand and discards the result. Then it evaluates and returns the right\n operand.\n\n Add support for comma expressions. Give them the same precedence and\n associativity as in C. Write the grammar, and then implement the necessary\n parsing code.\n\n2. Likewise, add support for the C-style conditional or \"ternary\" operator\n `?:`. What precedence level is allowed between the `?` and `:`? Is the whole\n operator left-associative or right-associative?\n\n3. Add error productions to handle each binary operator appearing without a\n left-hand operand. In other words, detect a binary operator appearing at the\n beginning of an expression. Report that as an error, but also parse and\n discard a right-hand operand with the appropriate precedence.\n\n[comma operator]: https://en.wikipedia.org/wiki/Comma_operator\n\n
\n\n
\n\n## Design Note: Logic Versus History\n\nLet's say we decide to add bitwise `&` and `|` operators to Lox. Where should we\nput them in the precedence hierarchy? C -- and most languages that follow in C's\nfootsteps -- place them below `==`. This is widely considered a mistake because\nit means common operations like testing a flag require parentheses.\n\n```c\nif (flags & FLAG_MASK == SOME_FLAG) { ... } // Wrong.\nif ((flags & FLAG_MASK) == SOME_FLAG) { ... } // Right.\n```\n\nShould we fix this for Lox and put bitwise operators higher up the precedence\ntable than C does? There are two strategies we can take.\n\nYou almost never want to use the result of an `==` expression as the operand to\na bitwise operator. By making bitwise bind tighter, users don't need to\nparenthesize as often. So if we do that, and users assume the precedence is\nchosen logically to minimize parentheses, they're likely to infer it correctly.\n\nThis kind of internal consistency makes the language easier to learn because\nthere are fewer edge cases and exceptions users have to stumble into and then\ncorrect. That's good, because before users can use our language, they have to\nload all of that syntax and semantics into their heads. A simpler, more rational\nlanguage *makes sense*.\n\nBut, for many users there is an even faster shortcut to getting our language's\nideas into their wetware -- *use concepts they already know*. Many newcomers to\nour language will be coming from some other language or languages. If our\nlanguage uses some of the same syntax or semantics as those, there is much less\nfor the user to learn (and *unlearn*).\n\nThis is particularly helpful with syntax. You may not remember it well today,\nbut way back when you learned your very first programming language, code\nprobably looked alien and unapproachable. Only through painstaking effort did\nyou learn to read and accept it. If you design a novel syntax for your new\nlanguage, you force users to start that process all over again.\n\nTaking advantage of what users already know is one of the most powerful tools\nyou can use to ease adoption of your language. It's almost impossible to\noverestimate how valuable this is. But it faces you with a nasty problem: What\nhappens when the thing the users all know *kind of sucks*? C's bitwise operator\nprecedence is a mistake that doesn't make sense. But it's a *familiar* mistake\nthat millions have already gotten used to and learned to live with.\n\nDo you stay true to your language's own internal logic and ignore history? Do\nyou start from a blank slate and first principles? Or do you weave your language\ninto the rich tapestry of programming history and give your users a leg up by\nstarting from something they already know?\n\nThere is no perfect answer here, only trade-offs. You and I are obviously biased\ntowards liking novel languages, so our natural inclination is to burn the\nhistory books and start our own story.\n\nIn practice, it's often better to make the most of what users already know.\nGetting them to come to your language requires a big leap. The smaller you can\nmake that chasm, the more people will be willing to cross it. But you can't\n*always* stick to history, or your language won't have anything new and\ncompelling to give people a *reason* to jump over.\n\n
\n" }, { "path": "book/representing-code.md", "content": "> To dwellers in a wood, almost every species of tree has its voice as well as\n> its feature.\n> Thomas Hardy, Under the Greenwood Tree\n\nIn the [last chapter][scanning], we took the raw source code as a string and\ntransformed it into a slightly higher-level representation: a series of tokens.\nThe parser we'll write in the [next chapter][parsing] takes those tokens and\ntransforms them yet again, into an even richer, more complex representation.\n\n[scanning]: scanning.html\n[parsing]: parsing-expressions.html\n\nBefore we can produce that representation, we need to define it. That's the\nsubject of this chapter. Along the way, we'll cover\nsome theory around formal grammars, feel the difference between functional and\nobject-oriented programming, go over a couple of design patterns, and do some\nmetaprogramming.\n\n\n\nBefore we do all that, let's focus on the main goal -- a representation for\ncode. It should be simple for the parser to produce and easy for the\ninterpreter to consume. If you haven't written a parser or interpreter yet,\nthose requirements aren't exactly illuminating. Maybe your intuition can help.\nWhat is your brain doing when you play the part of a *human* interpreter? How do\nyou mentally evaluate an arithmetic expression like this:\n\n```lox\n1 + 2 * 3 - 4\n```\n\nBecause you understand the order of operations -- the old \"[Please Excuse My\nDear Aunt Sally][sally]\" stuff -- you know that the multiplication is evaluated\nbefore the addition or subtraction. One way to visualize that precedence is\nusing a tree. Leaf nodes are numbers, and interior nodes are operators with\nbranches for each of their operands.\n\n[sally]: https://en.wikipedia.org/wiki/Order_of_operations#Mnemonics\n\nIn order to evaluate an arithmetic node, you need to know the numeric values of\nits subtrees, so you have to evaluate those first. That means working your way\nfrom the leaves up to the root -- a *post-order* traversal:\n\n\n\n\"Evaluating\n\n\n\nIf I gave you an arithmetic expression, you could draw one of these trees pretty\neasily. Given a tree, you can evaluate it without breaking a sweat. So it\nintuitively seems like a workable representation of our code is a tree that matches the grammatical structure -- the operator\nnesting -- of the language.\n\n\n\nWe need to get more precise about what that grammar is then. Like lexical\ngrammars in the last chapter, there is a long ton of theory around syntactic\ngrammars. We're going into that theory a little more than we did when scanning\nbecause it turns out to be a useful tool throughout much of the interpreter.\nWe start by moving one level up the [Chomsky hierarchy][]...\n\n[chomsky hierarchy]: https://en.wikipedia.org/wiki/Chomsky_hierarchy\n\n## Context-Free Grammars\n\nIn the last chapter, the formalism we used for defining the lexical grammar --\nthe rules for how characters get grouped into tokens -- was called a *regular\nlanguage*. That was fine for our scanner, which emits a flat sequence of tokens.\nBut regular languages aren't powerful enough to handle expressions which can\nnest arbitrarily deeply.\n\nWe need a bigger hammer, and that hammer is a **context-free grammar**\n(**CFG**). It's the next heaviest tool in the toolbox of\n**[formal grammars][]**. A formal grammar takes a set of atomic pieces it calls\nits \"alphabet\". Then it defines a (usually infinite) set of \"strings\" that are\n\"in\" the grammar. Each string is a sequence of \"letters\" in the alphabet.\n\n[formal grammars]: https://en.wikipedia.org/wiki/Formal_grammar\n\nI'm using all those quotes because the terms get a little confusing as you move\nfrom lexical to syntactic grammars. In our scanner's grammar, the alphabet\nconsists of individual characters and the strings are the valid lexemes --\nroughly \"words\". In the syntactic grammar we're talking about now, we're at a\ndifferent level of granularity. Now each \"letter\" in the alphabet is an entire\ntoken and a \"string\" is a sequence of *tokens* -- an entire expression.\n\nOof. Maybe a table will help:\n\n\n\n\n \n \n \n \n\n\n\n\n \n \n \n \n\n\n \n \n \n \n\n\n \n \n \n \n\n\n
TerminologyLexical grammarSyntactic grammar
The “alphabet” is . . .→ CharactersTokens
A “string” is . . .→ Lexeme or tokenExpression
It’s implemented by the . . .→ ScannerParser
\n\nA formal grammar's job is to specify which strings are valid and which aren't.\nIf we were defining a grammar for English sentences, \"eggs are tasty for\nbreakfast\" would be in the grammar, but \"tasty breakfast for are eggs\" would\nprobably not.\n\n### Rules for grammars\n\nHow do we write down a grammar that contains an infinite number of valid\nstrings? We obviously can't list them all out. Instead, we create a finite set\nof rules. You can think of them as a game that you can \"play\" in one of two\ndirections.\n\nIf you start with the rules, you can use them to *generate* strings that are in\nthe grammar. Strings created this way are called **derivations** because each is\n*derived* from the rules of the grammar. In each step of the game, you pick a\nrule and follow what it tells you to do. Most of the lingo around formal\ngrammars comes from playing them in this direction. Rules are called\n**productions** because they *produce* strings in the grammar.\n\nEach production in a context-free grammar has a **head** -- its name -- and a **body**, which describes what it generates. In\nits pure form, the body is simply a list of symbols. Symbols come in two\ndelectable flavors:\n\n\n\n* A **terminal** is a letter from the grammar's alphabet. You can think of it\n like a literal value. In the syntactic grammar we're defining, the terminals\n are individual lexemes -- tokens coming from the scanner like `if` or\n `1234`.\n\n These are called \"terminals\", in the sense of an \"end point\" because they\n don't lead to any further \"moves\" in the game. You simply produce that one\n symbol.\n\n* A **nonterminal** is a named reference to another rule in the grammar. It\n means \"play that rule and insert whatever it produces here\". In this way,\n the grammar composes.\n\nThere is one last refinement: you may have multiple rules with the same name.\nWhen you reach a nonterminal with that name, you are allowed to pick any of the\nrules for it, whichever floats your boat.\n\nTo make this concrete, we need a way to write down\nthese production rules. People have been trying to crystallize grammar all the\nway back to Pāṇini's *Ashtadhyayi*, which codified Sanskrit grammar a mere\ncouple thousand years ago. Not much progress happened until John Backus and\ncompany needed a notation for specifying ALGOL 58 and came up with\n[**Backus-Naur form**][bnf] (**BNF**). Since then, nearly everyone uses some\nflavor of BNF, tweaked to their own tastes.\n\n[bnf]: https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form\n\nI tried to come up with something clean. Each rule is a name, followed by an\narrow (`→`), followed by a sequence of symbols, and finally ending with a\nsemicolon (`;`). Terminals are quoted strings, and nonterminals are lowercase\nwords.\n\n\n\nUsing that, here's a grammar for breakfast menus:\n\n\n\n```ebnf\nbreakfast → protein \"with\" breakfast \"on the side\" ;\nbreakfast → protein ;\nbreakfast → bread ;\n\nprotein → crispiness \"crispy\" \"bacon\" ;\nprotein → \"sausage\" ;\nprotein → cooked \"eggs\" ;\n\ncrispiness → \"really\" ;\ncrispiness → \"really\" crispiness ;\n\ncooked → \"scrambled\" ;\ncooked → \"poached\" ;\ncooked → \"fried\" ;\n\nbread → \"toast\" ;\nbread → \"biscuits\" ;\nbread → \"English muffin\" ;\n```\n\nWe can use this grammar to generate random breakfasts. Let's play a round and\nsee how it works. By age-old convention, the game starts with the first rule in\nthe grammar, here `breakfast`. There are three productions for that, and we\nrandomly pick the first one. Our resulting string looks like:\n\n```text\nprotein \"with\" breakfast \"on the side\"\n```\n\nWe need to expand that first nonterminal, `protein`, so we pick a production for\nthat. Let's pick:\n\n```ebnf\nprotein → cooked \"eggs\" ;\n```\n\nNext, we need a production for `cooked`, and so we pick `\"poached\"`. That's a\nterminal, so we add that. Now our string looks like:\n\n```text\n\"poached\" \"eggs\" \"with\" breakfast \"on the side\"\n```\n\nThe next non-terminal is `breakfast` again. The first `breakfast` production we\nchose recursively refers back to the `breakfast` rule. Recursion in the grammar\nis a good sign that the language being defined is context-free instead of\nregular. In particular, recursion where the recursive nonterminal has\nproductions on both sides implies that the language is\nnot regular.\n\n\n\nWe could keep picking the first production for `breakfast` over and over again\nyielding all manner of breakfasts like \"bacon with sausage with scrambled eggs\nwith bacon...\" We won't though. This time we'll pick `bread`. There are three\nrules for that, each of which contains only a terminal. We'll pick \"English\nmuffin\".\n\nWith that, every nonterminal in the string has been expanded until it finally\ncontains only terminals and we're left with:\n\n\"Playing\" the grammar to generate a string.\n\nThrow in some ham and Hollandaise, and you've got eggs Benedict.\n\nAny time we hit a rule that had multiple productions, we just picked one\narbitrarily. It is this flexibility that allows a short number of grammar rules\nto encode a combinatorially larger set of strings. The fact that a rule can\nrefer to itself -- directly or indirectly -- kicks it up even more, letting us\npack an infinite number of strings into a finite grammar.\n\n### Enhancing our notation\n\nStuffing an infinite set of strings in a handful of rules is pretty fantastic,\nbut let's take it further. Our notation works, but it's tedious. So, like any\ngood language designer, we'll sprinkle a little syntactic sugar on top -- some\nextra convenience notation. In addition to terminals and nonterminals, we'll\nallow a few other kinds of expressions in the body of a rule:\n\n* Instead of repeating the rule name each time we want to add another\n production for it, we'll allow a series of productions separated by a pipe\n (`|`).\n\n ```ebnf\n bread → \"toast\" | \"biscuits\" | \"English muffin\" ;\n ```\n\n* Further, we'll allow parentheses for grouping and then allow `|` within that\n to select one from a series of options within the middle of a production.\n\n ```ebnf\n protein → ( \"scrambled\" | \"poached\" | \"fried\" ) \"eggs\" ;\n ```\n\n* Using recursion to support repeated sequences of symbols has a certain\n appealing purity, but it's kind of a chore to\n make a separate named sub-rule each time we want to loop. So, we also use a\n postfix `*` to allow the previous symbol or group to be repeated zero or\n more times.\n\n ```ebnf\n crispiness → \"really\" \"really\"* ;\n ```\n\n\n\n* A postfix `+` is similar, but requires the preceding production to appear\n at least once.\n\n ```ebnf\n crispiness → \"really\"+ ;\n ```\n\n* A postfix `?` is for an optional production. The thing before it can appear\n zero or one time, but not more.\n\n ```ebnf\n breakfast → protein ( \"with\" breakfast \"on the side\" )? ;\n ```\n\nWith all of those syntactic niceties, our breakfast grammar condenses down to:\n\n```ebnf\nbreakfast → protein ( \"with\" breakfast \"on the side\" )?\n | bread ;\n\nprotein → \"really\"+ \"crispy\" \"bacon\"\n | \"sausage\"\n | ( \"scrambled\" | \"poached\" | \"fried\" ) \"eggs\" ;\n\nbread → \"toast\" | \"biscuits\" | \"English muffin\" ;\n```\n\nNot too bad, I hope. If you're used to grep or using [regular\nexpressions][regex] in your text editor, most of the punctuation should be\nfamiliar. The main difference is that symbols here represent entire tokens, not\nsingle characters.\n\n[regex]: https://en.wikipedia.org/wiki/Regular_expression#Standards\n\nWe'll use this notation throughout the rest of the book to precisely describe\nLox's grammar. As you work on programming languages, you'll find that\ncontext-free grammars (using this or [EBNF][] or some other notation) help you\ncrystallize your informal syntax design ideas. They are also a handy medium for\ncommunicating with other language hackers about syntax.\n\n[ebnf]: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form\n\nThe rules and productions we define for Lox are also our guide to the tree data\nstructure we're going to implement to represent code in memory. Before we can do\nthat, we need an actual grammar for Lox, or at least enough of one for us to get\nstarted.\n\n### A Grammar for Lox expressions\n\nIn the previous chapter, we did Lox's entire lexical grammar in one fell swoop.\nEvery keyword and bit of punctuation is there. The syntactic grammar is larger,\nand it would be a real bore to grind through the entire thing before we actually\nget our interpreter up and running.\n\nInstead, we'll crank through a subset of the language in the next couple of\nchapters. Once we have that mini-language represented, parsed, and interpreted,\nthen later chapters will progressively add new features to it, including the new\nsyntax. For now, we are going to worry about only a handful of expressions:\n\n* **Literals.** Numbers, strings, Booleans, and `nil`.\n\n* **Unary expressions.** A prefix `!` to perform a logical not, and `-` to\n negate a number.\n\n* **Binary expressions.** The infix arithmetic (`+`, `-`, `*`, `/`) and logic\n operators (`==`, `!=`, `<`, `<=`, `>`, `>=`) we know and love.\n\n* **Parentheses.** A pair of `(` and `)` wrapped around an expression.\n\nThat gives us enough syntax for expressions like:\n\n```lox\n1 - (2 * 3) < 4 == false\n```\n\nUsing our handy dandy new notation, here's a grammar for those:\n\n```ebnf\nexpression → literal\n | unary\n | binary\n | grouping ;\n\nliteral → NUMBER | STRING | \"true\" | \"false\" | \"nil\" ;\ngrouping → \"(\" expression \")\" ;\nunary → ( \"-\" | \"!\" ) expression ;\nbinary → expression operator expression ;\noperator → \"==\" | \"!=\" | \"<\" | \"<=\" | \">\" | \">=\"\n | \"+\" | \"-\" | \"*\" | \"/\" ;\n```\n\nThere's one bit of extra metasyntax here. In addition\nto quoted strings for terminals that match exact lexemes, we `CAPITALIZE`\nterminals that are a single lexeme whose text representation may vary. `NUMBER`\nis any number literal, and `STRING` is any string literal. Later, we'll do the\nsame for `IDENTIFIER`.\n\nThis grammar is actually ambiguous, which we'll see when we get to parsing it.\nBut it's good enough for now.\n\n\n\n## Implementing Syntax Trees\n\nFinally, we get to write some code. That little expression grammar is our\nskeleton. Since the grammar is recursive -- note how `grouping`, `unary`, and\n`binary` all refer back to `expression` -- our data structure will form a tree.\nSince this structure represents the syntax of our language, it's called a **syntax tree**.\n\n\n\nOur scanner used a single Token class to represent all kinds of lexemes. To\ndistinguish the different kinds -- think the number `123` versus the string\n`\"123\"` -- we included a simple TokenType enum. Syntax trees are not so homogeneous. Unary expressions have a single operand,\nbinary expressions have two, and literals have none.\n\nWe *could* mush that all together into a single Expression class with an\narbitrary list of children. Some compilers do. But I like getting the most out\nof Java's type system. So we'll define a base class for expressions. Then, for\neach kind of expression -- each production under `expression` -- we create a\nsubclass that has fields for the nonterminals specific to that rule. This way,\nwe get a compile error if we, say, try to access the second operand of a unary\nexpression.\n\n\n\nSomething like this:\n\n```java\npackage com.craftinginterpreters.lox;\n\nabstract class Expr { // [expr]\n static class Binary extends Expr {\n Binary(Expr left, Token operator, Expr right) {\n this.left = left;\n this.operator = operator;\n this.right = right;\n }\n\n final Expr left;\n final Token operator;\n final Expr right;\n }\n\n // Other expressions...\n}\n```\n\n\n\nExpr is the base class that all expression classes inherit from. As you can see\nfrom `Binary`, the subclasses are nested inside of it. There's no technical need\nfor this, but it lets us cram all of the classes into a single Java file.\n\n### Disoriented objects\n\nYou'll note that, much like the Token class, there aren't any methods here. It's\na dumb structure. Nicely typed, but merely a bag of data. This feels strange in\nan object-oriented language like Java. Shouldn't the class *do stuff*?\n\nThe problem is that these tree classes aren't owned by any single domain. Should\nthey have methods for parsing since that's where the trees are created? Or\ninterpreting since that's where they are consumed? Trees span the border between\nthose territories, which means they are really owned by *neither*.\n\nIn fact, these types exist to enable the parser and interpreter to\n*communicate*. That lends itself to types that are simply data with no\nassociated behavior. This style is very natural in functional languages like\nLisp and ML where *all* data is separate from behavior, but it feels odd in\nJava.\n\nFunctional programming aficionados right now are jumping up to exclaim \"See!\nObject-oriented languages are a bad fit for an interpreter!\" I won't go that\nfar. You'll recall that the scanner itself was admirably suited to\nobject-orientation. It had all of the mutable state to keep track of where it\nwas in the source code, a well-defined set of public methods, and a handful of\nprivate helpers.\n\nMy feeling is that each phase or part of the interpreter works fine in an\nobject-oriented style. It is the data structures that flow between them that are\nstripped of behavior.\n\n### Metaprogramming the trees\n\nJava can express behavior-less classes, but I wouldn't say that it's\nparticularly great at it. Eleven lines of code to stuff three fields in an\nobject is pretty tedious, and when we're all done, we're going to have 21 of\nthese classes.\n\nI don't want to waste your time or my ink writing all that down. Really, what is\nthe essence of each subclass? A name, and a list of typed fields. That's it.\nWe're smart language hackers, right? Let's automate.\n\n\n\nInstead of tediously handwriting each class definition, field declaration,\nconstructor, and initializer, we'll hack together a script that does it for us. It has a description of each\ntree type -- its name and fields -- and it prints out the Java code needed to\ndefine a class with that name and state.\n\nThis script is a tiny Java command-line app that generates a file named\n\"Expr.java\":\n\n\n\n^code generate-ast\n\nNote that this file is in a different package, `.tool` instead of `.lox`. This\nscript isn't part of the interpreter itself. It's a tool *we*, the people\nhacking on the interpreter, run ourselves to generate the syntax tree classes.\nWhen it's done, we treat \"Expr.java\" like any other file in the implementation.\nWe are merely automating how that file gets authored.\n\nTo generate the classes, it needs to have some description of each type and its\nfields.\n\n^code call-define-ast (1 before, 1 after)\n\nFor brevity's sake, I jammed the descriptions of the expression types into\nstrings. Each is the name of the class followed by `:` and the list of fields,\nseparated by commas. Each field has a type and a name.\n\nThe first thing `defineAst()` needs to do is output the base Expr class.\n\n^code define-ast\n\nWhen we call this, `baseName` is \"Expr\", which is both the name of the class and\nthe name of the file it outputs. We pass this as an argument instead of\nhardcoding the name because we'll add a separate family of classes later for\nstatements.\n\nInside the base class, we define each subclass.\n\n^code nested-classes (2 before, 1 after)\n\n\n\nThat code, in turn, calls:\n\n^code define-type\n\nThere we go. All of that glorious Java boilerplate is done. It declares each\nfield in the class body. It defines a constructor for the class with parameters\nfor each field and initializes them in the body.\n\nCompile and run this Java program now and it blasts\nout a new “.java\" file containing a few dozen lines of code. That file's\nabout to get even longer.\n\n\n\n## Working with Trees\n\nPut on your imagination hat for a moment. Even though we aren't there yet,\nconsider what the interpreter will do with the syntax trees. Each kind of\nexpression in Lox behaves differently at runtime. That means the interpreter\nneeds to select a different chunk of code to handle each expression type. With\ntokens, we can simply switch on the TokenType. But we don't have a \"type\" enum\nfor the syntax trees, just a separate Java class for each one.\n\nWe could write a long chain of type tests:\n\n```java\nif (expr instanceof Expr.Binary) {\n // ...\n} else if (expr instanceof Expr.Grouping) {\n // ...\n} else // ...\n```\n\nBut all of those sequential type tests are slow. Expression types whose names\nare alphabetically later would take longer to execute because they'd fall\nthrough more `if` cases before finding the right type. That's not my idea of an\nelegant solution.\n\nWe have a family of classes and we need to associate a chunk of behavior with\neach one. The natural solution in an object-oriented language like Java is to\nput those behaviors into methods on the classes themselves. We could add an\nabstract `interpret()` method on Expr\nwhich each subclass would then implement to interpret itself.\n\n\n\nThis works alright for tiny projects, but it scales poorly. Like I noted before,\nthese tree classes span a few domains. At the very least, both the parser and\ninterpreter will mess with them. As [you'll see later][resolution], we need to\ndo name resolution on them. If our language was statically typed, we'd have a\ntype checking pass.\n\n[resolution]: resolving-and-binding.html\n\nIf we added instance methods to the expression classes for every one of those\noperations, that would smush a bunch of different domains together. That\nviolates [separation of concerns][] and leads to hard-to-maintain code.\n\n[separation of concerns]: https://en.wikipedia.org/wiki/Separation_of_concerns\n\n### The expression problem\n\nThis problem is more fundamental than it may seem at first. We have a handful of\ntypes, and a handful of high-level operations like \"interpret\". For each pair of\ntype and operation, we need a specific implementation. Picture a table:\n\n\"A\n\nRows are types, and columns are operations. Each cell represents the unique\npiece of code to implement that operation on that type.\n\nAn object-oriented language like Java assumes that all of the code in one row\nnaturally hangs together. It figures all the things you do with a type are\nlikely related to each other, and the language makes it easy to define them\ntogether as methods inside the same class.\n\n\"The\n\nThis makes it easy to extend the table by adding new rows. Simply define a new\nclass. No existing code has to be touched. But imagine if you want to add a new\n*operation* -- a new column. In Java, that means cracking open each of those\nexisting classes and adding a method to it.\n\nFunctional paradigm languages in the ML family flip that\naround. There, you don't have classes with methods. Types and functions are\ntotally distinct. To implement an operation for a number of different types, you\ndefine a single function. In the body of that function, you use *pattern\nmatching* -- sort of a type-based switch on steroids -- to implement the\noperation for each type all in one place.\n\n\n\nThis makes it trivial to add new operations -- simply define another function\nthat pattern matches on all of the types.\n\n\"The\n\nBut, conversely, adding a new type is hard. You have to go back and add a new\ncase to all of the pattern matches in all of the existing functions.\n\nEach style has a certain \"grain\" to it. That's what the paradigm name literally\nsays -- an object-oriented language wants you to *orient* your code along the\nrows of types. A functional language instead encourages you to lump each\ncolumn's worth of code together into a *function*.\n\nA bunch of smart language nerds noticed that neither style made it easy to add\n*both* rows and columns to the table. They called this\ndifficulty the \"expression problem\" because -- like we are now -- they first ran\ninto it when they were trying to figure out the best way to model expression\nsyntax tree nodes in a compiler.\n\n\n\nPeople have thrown all sorts of language features, design patterns, and\nprogramming tricks to try to knock that problem down but no perfect language has\nfinished it off yet. In the meantime, the best we can do is try to pick a\nlanguage whose orientation matches the natural architectural seams in the\nprogram we're writing.\n\nObject-orientation works fine for many parts of our interpreter, but these tree\nclasses rub against the grain of Java. Fortunately, there's a design pattern we\ncan bring to bear on it.\n\n### The Visitor pattern\n\nThe **Visitor pattern** is the most widely misunderstood pattern in all of\n*Design Patterns*, which is really saying something when you look at the\nsoftware architecture excesses of the past couple of decades.\n\nThe trouble starts with terminology. The pattern isn't about \"visiting\", and the\n\"accept\" method in it doesn't conjure up any helpful imagery either. Many think\nthe pattern has to do with traversing trees, which isn't the case at all. We\n*are* going to use it on a set of classes that are tree-like, but that's a\ncoincidence. As you'll see, the pattern works as well on a single object.\n\nThe Visitor pattern is really about approximating the functional style within an\nOOP language. It lets us add new columns to that table easily. We can define all\nof the behavior for a new operation on a set of types in one place, without\nhaving to touch the types themselves. It does this the same way we solve almost\nevery problem in computer science: by adding a layer of indirection.\n\nBefore we apply it to our auto-generated Expr classes, let's walk through a\nsimpler example. Say we have two kinds of pastries: beignets and crullers.\n\n\n\n^code pastries (no location)\n\nWe want to be able to define new pastry operations -- cooking them, eating them,\ndecorating them, etc. -- without having to add a new method to each class every\ntime. Here's how we do it. First, we define a separate interface.\n\n^code pastry-visitor (no location)\n\n\n\nEach operation that can be performed on pastries is a new class that implements\nthat interface. It has a concrete method for each type of pastry. That keeps the\ncode for the operation on both types all nestled snugly together in one class.\n\nGiven some pastry, how do we route it to the correct method on the visitor based\non its type? Polymorphism to the rescue! We add this method to Pastry:\n\n^code pastry-accept (1 before, 1 after, no location)\n\nEach subclass implements it.\n\n^code beignet-accept (1 before, 1 after, no location)\n\nAnd:\n\n^code cruller-accept (1 before, 1 after, no location)\n\nTo perform an operation on a pastry, we call its `accept()` method and pass in\nthe visitor for the operation we want to execute. The pastry -- the specific\nsubclass's overriding implementation of `accept()` -- turns around and calls the\nappropriate visit method on the visitor and passes *itself* to it.\n\nThat's the heart of the trick right there. It lets us use polymorphic dispatch\non the *pastry* classes to select the appropriate method on the *visitor* class.\nIn the table, each pastry class is a row, but if you look at all of the methods\nfor a single visitor, they form a *column*.\n\n\"Now\n\nWe added one `accept()` method to each class, and we can use it for as many\nvisitors as we want without ever having to touch the pastry classes again. It's\na clever pattern.\n\n### Visitors for expressions\n\nOK, let's weave it into our expression classes. We'll also refine the pattern a little. In the pastry example, the\nvisit and `accept()` methods don't return anything. In practice, visitors often\nwant to define operations that produce values. But what return type should\n`accept()` have? We can't assume every visitor class wants to produce the same\ntype, so we'll use generics to let each implementation fill in a return type.\n\n\n\nFirst, we define the visitor interface. Again, we nest it inside the base class\nso that we can keep everything in one file.\n\n^code call-define-visitor (2 before, 1 after)\n\nThat function generates the visitor interface.\n\n^code define-visitor\n\nHere, we iterate through all of the subclasses and declare a visit method for\neach one. When we define new expression types later, this will automatically\ninclude them.\n\nInside the base class, we define the abstract `accept()` method.\n\n^code base-accept-method (2 before, 1 after)\n\nFinally, each subclass implements that and calls the right visit method for its\nown type.\n\n^code accept-method (1 before, 2 after)\n\nThere we go. Now we can define operations on expressions without having to muck\nwith the classes or our generator script. Compile and run this generator script\nto output an updated \"Expr.java\" file. It contains a generated Visitor\ninterface and a set of expression node classes that support the Visitor pattern\nusing it.\n\nBefore we end this rambling chapter, let's implement that Visitor interface and\nsee the pattern in action.\n\n## A (Not Very) Pretty Printer\n\nWhen we debug our parser and interpreter, it's often useful to look at a parsed\nsyntax tree and make sure it has the structure we expect. We could inspect it in\nthe debugger, but that can be a chore.\n\nInstead, we'd like some code that, given a syntax tree, produces an unambiguous\nstring representation of it. Converting a tree to a string is sort of the\nopposite of a parser, and is often called \"pretty printing\" when the goal is to\nproduce a string of text that is valid syntax in the source language.\n\nThat's not our goal here. We want the string to very explicitly show the nesting\nstructure of the tree. A printer that returned `1 + 2 * 3` isn't super helpful\nif what we're trying to debug is whether operator precedence is handled\ncorrectly. We want to know if the `+` or `*` is at the top of the tree.\n\nTo that end, the string representation we produce isn't going to be Lox syntax.\nInstead, it will look a lot like, well, Lisp. Each expression is explicitly\nparenthesized, and all of its subexpressions and tokens are contained in that.\n\nGiven a syntax tree like:\n\n\"An\n\nIt produces:\n\n```text\n(* (- 123) (group 45.67))\n```\n\nNot exactly \"pretty\", but it does show the nesting and grouping explicitly. To\nimplement this, we define a new class.\n\n^code ast-printer\n\nAs you can see, it implements the visitor interface. That means we need visit\nmethods for each of the expression types we have so far.\n\n^code visit-methods (2 before, 1 after)\n\nLiteral expressions are easy -- they convert the value to a string with a little\ncheck to handle Java's `null` standing in for Lox's `nil`. The other expressions\nhave subexpressions, so they use this `parenthesize()` helper method:\n\n^code print-utilities\n\nIt takes a name and a list of subexpressions and wraps them all up in\nparentheses, yielding a string like:\n\n```text\n(+ 1 2)\n```\n\nNote that it calls `accept()` on each subexpression and passes in itself. This\nis the recursive step that lets us print an entire\ntree.\n\n\n\nWe don't have a parser yet, so it's hard to see this in action. For now, we'll\nhack together a little `main()` method that manually instantiates a tree and\nprints it.\n\n^code printer-main\n\nIf we did everything right, it prints:\n\n```text\n(* (- 123) (group 45.67))\n```\n\nYou can go ahead and delete this method. We won't need it. Also, as we add new\nsyntax tree types, I won't bother showing the necessary visit methods for them\nin AstPrinter. If you want to (and you want the Java compiler to not yell at\nyou), go ahead and add them yourself. It will come in handy in the next chapter\nwhen we start parsing Lox code into syntax trees. Or, if you don't care to\nmaintain AstPrinter, feel free to delete it. We won't need it again.\n\n
\n\n## Challenges\n\n1. Earlier, I said that the `|`, `*`, and `+` forms we added to our grammar\n metasyntax were just syntactic sugar. Take this grammar:\n\n ```ebnf\n expr → expr ( \"(\" ( expr ( \",\" expr )* )? \")\" | \".\" IDENTIFIER )+\n | IDENTIFIER\n | NUMBER\n ```\n\n Produce a grammar that matches the same language but does not use any of\n that notational sugar.\n\n *Bonus:* What kind of expression does this bit of grammar encode?\n\n1. The Visitor pattern lets you emulate the functional style in an\n object-oriented language. Devise a complementary pattern for a functional\n language. It should let you bundle all of the operations on one type\n together and let you define new types easily.\n\n (SML or Haskell would be ideal for this exercise, but Scheme or another Lisp\n works as well.)\n\n1. In [reverse Polish notation][rpn] (RPN), the operands to an arithmetic\n operator are both placed before the operator, so `1 + 2` becomes `1 2 +`.\n Evaluation proceeds from left to right. Numbers are pushed onto an implicit\n stack. An arithmetic operator pops the top two numbers, performs the\n operation, and pushes the result. Thus, this:\n\n ```lox\n (1 + 2) * (4 - 3)\n ```\n\n in RPN becomes:\n\n ```lox\n 1 2 + 4 3 - *\n ```\n\n Define a visitor class for our syntax tree classes that takes an expression,\n converts it to RPN, and returns the resulting string.\n\n[rpn]: https://en.wikipedia.org/wiki/Reverse_Polish_notation\n\n
\n" }, { "path": "book/resolving-and-binding.md", "content": "> Once in a while you find yourself in an odd situation. You get into it by\n> degrees and in the most natural way but, when you are right in the midst of\n> it, you are suddenly astonished and ask yourself how in the world it all came\n> about.\n>\n> Thor Heyerdahl, Kon-Tiki\n\nOh, no! Our language implementation is taking on water! Way back when we [added\nvariables and blocks][statements], we had scoping nice and tight. But when we\n[later added closures][functions], a hole opened in our formerly waterproof\ninterpreter. Most real programs are unlikely to slip through this hole, but as\nlanguage implementers, we take a sacred vow to care about correctness even in\nthe deepest, dampest corners of the semantics.\n\n[statements]: statements-and-state.html\n[functions]: functions.html\n\nWe will spend this entire chapter exploring that leak, and then carefully\npatching it up. In the process, we will gain a more rigorous understanding of\nlexical scoping as used by Lox and other languages in the C tradition. We'll\nalso get a chance to learn about *semantic analysis* -- a powerful technique for\nextracting meaning from the user's source code without having to run it.\n\n## Static Scope\n\nA quick refresher: Lox, like most modern languages, uses *lexical* scoping. This\nmeans that you can figure out which declaration a variable name refers to just\nby reading the text of the program. For example:\n\n```lox\nvar a = \"outer\";\n{\n var a = \"inner\";\n print a;\n}\n```\n\nHere, we know that the `a` being printed is the variable declared on the\nprevious line, and not the global one. Running the program doesn't -- *can't* --\naffect this. The scope rules are part of the *static* semantics of the language,\nwhich is why they're also called *static scope*.\n\nI haven't spelled out those scope rules, but now is the time for precision:\n\n\n\n**A variable usage refers to the preceding declaration with the same name in the\ninnermost scope that encloses the expression where the variable is used.**\n\nThere's a lot to unpack in that:\n\n* I say \"variable usage\" instead of \"variable expression\" to cover both\n variable expressions and assignments. Likewise with \"expression where the\n variable is used\".\n\n* \"Preceding\" means appearing before *in the program text*.\n\n ```lox\n var a = \"outer\";\n {\n print a;\n var a = \"inner\";\n }\n ```\n\n Here, the `a` being printed is the outer one since it appears before the `print` statement that uses it. In most\n cases, in straight line code, the declaration preceding in *text* will also\n precede the usage in *time*. But that's not always true. As we'll see,\n functions may defer a chunk of code such that its *dynamic temporal*\n execution no longer mirrors the *static textual* ordering.\n\n \n\n* \"Innermost\" is there because of our good friend shadowing. There may be more\n than one variable with the given name in enclosing scopes, as in:\n\n ```lox\n var a = \"outer\";\n {\n var a = \"inner\";\n print a;\n }\n ```\n\n Our rule disambiguates this case by saying the innermost scope wins.\n\nSince this rule makes no mention of any runtime behavior, it implies that a\nvariable expression always refers to the same declaration through the entire\nexecution of the program. Our interpreter so far *mostly* implements the rule\ncorrectly. But when we added closures, an error snuck in.\n\n```lox\nvar a = \"global\";\n{\n fun showA() {\n print a;\n }\n\n showA();\n var a = \"block\";\n showA();\n}\n```\n\nBefore you type this in and run it, decide what you\nthink it *should* print.\n\n\n\nOK... got it? If you're familiar with closures in other languages, you'll expect\nit to print \"global\" twice. The first call to `showA()` should definitely print\n\"global\" since we haven't even reached the declaration of the inner `a` yet. And\nby our rule that a variable expression always resolves to the same variable,\nthat implies the second call to `showA()` should print the same thing.\n\nAlas, it prints:\n\n```text\nglobal\nblock\n```\n\nLet me stress that this program never reassigns any variable and contains only a\nsingle `print` statement. Yet, somehow, that `print` statement for a\nnever-assigned variable prints two different values at different points in time.\nWe definitely broke something somewhere.\n\n### Scopes and mutable environments\n\nIn our interpreter, environments are the dynamic manifestation of static scopes.\nThe two mostly stay in sync with each other -- we create a new environment when\nwe enter a new scope, and discard it when we leave the scope. There is one other\noperation we perform on environments: binding a variable in one. This is where\nour bug lies.\n\nLet's walk through that problematic example and see what the environments look\nlike at each step. First, we declare `a` in the global scope.\n\n\"The\n\nThat gives us a single environment with a single variable in it. Then we enter\nthe block and execute the declaration of `showA()`.\n\n\"A\n\nWe get a new environment for the block. In that, we declare one name, `showA`,\nwhich is bound to the LoxFunction object we create to represent the function.\nThat object has a `closure` field that captures the environment where the\nfunction was declared, so it has a reference back to the environment for the\nblock.\n\nNow we call `showA()`.\n\n\"An\n\nThe interpreter dynamically creates a new environment for the function body of\n`showA()`. It's empty since that function doesn't declare any variables. The\nparent of that environment is the function's closure -- the outer block\nenvironment.\n\nInside the body of `showA()`, we print the value of `a`. The interpreter looks\nup this value by walking the chain of environments. It gets all the way\nto the global environment before finding it there and printing `\"global\"`.\nGreat.\n\nNext, we declare the second `a`, this time inside the block.\n\n\"The\n\nIt's in the same block -- the same scope -- as `showA()`, so it goes into the\nsame environment, which is also the same environment `showA()`'s closure refers\nto. This is where it gets interesting. We call `showA()` again.\n\n\"An\n\nWe create a new empty environment for the body of `showA()` again, wire it up to\nthat closure, and run the body. When the interpreter walks the chain of\nenvironments to find `a`, it now discovers the *new* `a` in the block\nenvironment. Boo.\n\nI chose to implement environments in a way that I hoped would agree with your\ninformal intuition around scopes. We tend to consider all of the code within a\nblock as being within the same scope, so our interpreter uses a single\nenvironment to represent that. Each environment is a mutable hash table. When a\nnew local variable is declared, it gets added to the existing environment for\nthat scope.\n\nThat intuition, like many in life, isn't quite right. A block is not necessarily\nall the same scope. Consider:\n\n```lox\n{\n var a;\n // 1.\n var b;\n // 2.\n}\n```\n\nAt the first marked line, only `a` is in scope. At the second line, both `a` and\n`b` are. If you define a \"scope\" to be a set of declarations, then those are\nclearly not the same scope -- they don't contain the same declarations. It's\nlike each `var` statement splits the block into two\nseparate scopes, the scope before the variable is declared and the one after,\nwhich includes the new variable.\n\n\n\nBut in our implementation, environments do act like the entire block is one\nscope, just a scope that changes over time. Closures do not like that. When a\nfunction is declared, it captures a reference to the current environment. The\nfunction *should* capture a frozen snapshot of the environment *as it existed at\nthe moment the function was declared*. But instead, in the Java code, it has a\nreference to the actual mutable environment object. When a variable is later\ndeclared in the scope that environment corresponds to, the closure sees the new\nvariable, even though the declaration does *not* precede the function.\n\n### Persistent environments\n\nThere is a style of programming that uses what are called **persistent data\nstructures**. Unlike the squishy data structures you're familiar with in\nimperative programming, a persistent data structure can never be directly\nmodified. Instead, any \"modification\" to an existing structure produces a brand new object that contains all of the original data and\nthe new modification. The original is left unchanged.\n\n\n\nIf we were to apply that technique to Environment, then every time you declared\na variable it would return a *new* environment that contained all of the\npreviously declared variables along with the one new name. Declaring a variable\nwould do the implicit \"split\" where you have an environment before the variable\nis declared and one after:\n\n\"Separate\n\nA closure retains a reference to the Environment instance in play when the\nfunction was declared. Since any later declarations in that block would produce\nnew Environment objects, the closure wouldn't see the new variables and our bug\nwould be fixed.\n\nThis is a legit way to solve the problem, and it's the classic way to implement\nenvironments in Scheme interpreters. We could do that for Lox, but it would mean\ngoing back and changing a pile of existing code.\n\nI won't drag you through that. We'll keep the way we represent environments the\nsame. Instead of making the data more statically structured, we'll bake the\nstatic resolution into the access *operation* itself.\n\n## Semantic Analysis\n\nOur interpreter **resolves** a variable -- tracks down which declaration it\nrefers to -- each and every time the variable expression is evaluated. If that\nvariable is swaddled inside a loop that runs a thousand times, that variable\ngets re-resolved a thousand times.\n\nWe know static scope means that a variable usage always resolves to the same\ndeclaration, which can be determined just by looking at the text. Given that,\nwhy are we doing it dynamically every time? Doing so doesn't just open the hole\nthat leads to our annoying bug, it's also needlessly slow.\n\nA better solution is to resolve each variable use *once*. Write a chunk of code\nthat inspects the user's program, finds every variable mentioned, and figures\nout which declaration each refers to. This process is an example of a **semantic\nanalysis**. Where a parser tells only if a program is grammatically correct (a\n*syntactic* analysis), semantic analysis goes farther and starts to figure out\nwhat pieces of the program actually mean. In this case, our analysis will\nresolve variable bindings. We'll know not just that an expression *is* a\nvariable, but *which* variable it is.\n\nThere are a lot of ways we could store the binding between a variable and its\ndeclaration. When we get to the C interpreter for Lox, we'll have a *much* more\nefficient way of storing and accessing local variables. But for jlox, I want to\nminimize the collateral damage we inflict on our existing codebase. I'd hate to\nthrow out a bunch of mostly fine code.\n\nInstead, we'll store the resolution in a way that makes the most out of our\nexisting Environment class. Recall how the accesses of `a` are interpreted in\nthe problematic example.\n\n\"An\n\nIn the first (correct) evaluation, we look at three environments in the chain\nbefore finding the global declaration of `a`. Then, when the inner `a` is later\ndeclared in a block scope, it shadows the global one.\n\n\"An\n\nThe next lookup walks the chain, finds `a` in the *second* environment and\nstops there. Each environment corresponds to a single lexical scope where\nvariables are declared. If we could ensure a variable lookup always walked the\n*same* number of links in the environment chain, that would ensure that it\nfound the same variable in the same scope every time.\n\nTo \"resolve\" a variable usage, we only need to calculate how many \"hops\" away\nthe declared variable will be in the environment chain. The interesting question\nis *when* to do this calculation -- or, put differently, where in our\ninterpreter's implementation do we stuff the code for it?\n\nSince we're calculating a static property based on the structure of the source\ncode, the obvious answer is in the parser. That is the traditional home, and is\nwhere we'll put it later in clox. It would work here too, but I want an excuse to\nshow you another technique. We'll write our resolver as a separate pass.\n\n### A variable resolution pass\n\nAfter the parser produces the syntax tree, but before the interpreter starts\nexecuting it, we'll do a single walk over the tree to resolve all of the\nvariables it contains. Additional passes between parsing and execution are\ncommon. If Lox had static types, we could slide a type checker in there.\nOptimizations are often implemented in separate passes like this too. Basically,\nany work that doesn't rely on state that's only available at runtime can be done\nin this way.\n\nOur variable resolution pass works like a sort of mini-interpreter. It walks the\ntree, visiting each node, but a static analysis is different from a dynamic\nexecution:\n\n* **There are no side effects.** When the static analysis visits a print\n statement, it doesn't actually print anything. Calls to native functions or\n other operations that reach out to the outside world are stubbed out and\n have no effect.\n\n* **There is no control flow.** Loops are visited only once. Both branches are visited in `if` statements. Logic\n operators are not short-circuited.\n\n\n\n## A Resolver Class\n\nLike everything in Java, our variable resolution pass is embodied in a class.\n\n^code resolver\n\nSince the resolver needs to visit every node in the syntax tree, it implements\nthe visitor abstraction we already have in place. Only a few kinds of nodes are\ninteresting when it comes to resolving variables:\n\n* A block statement introduces a new scope for the statements it contains.\n\n* A function declaration introduces a new scope for its body and binds its\n parameters in that scope.\n\n* A variable declaration adds a new variable to the current scope.\n\n* Variable and assignment expressions need to have their variables resolved.\n\nThe rest of the nodes don't do anything special, but we still need to implement\nvisit methods for them that traverse into their subtrees. Even though a `+`\nexpression doesn't *itself* have any variables to resolve, either of its\noperands might.\n\n### Resolving blocks\n\nWe start with blocks since they create the local scopes where all the magic\nhappens.\n\n^code visit-block-stmt\n\nThis begins a new scope, traverses into the statements inside the block, and\nthen discards the scope. The fun stuff lives in those helper methods. We start\nwith the simple one.\n\n^code resolve-statements\n\nThis walks a list of statements and resolves each one. It in turn calls:\n\n^code resolve-stmt\n\nWhile we're at it, let's add another overload that we'll need later for\nresolving an expression.\n\n^code resolve-expr\n\nThese methods are similar to the `evaluate()` and `execute()` methods in\nInterpreter -- they turn around and apply the Visitor pattern to the given\nsyntax tree node.\n\nThe real interesting behavior is around scopes. A new block scope is created\nlike so:\n\n^code begin-scope\n\nLexical scopes nest in both the interpreter and the resolver. They behave like a\nstack. The interpreter implements that stack using a linked list -- the chain of\nEnvironment objects. In the resolver, we use an actual Java Stack.\n\n^code scopes-field (1 before, 2 after)\n\nThis field keeps track of the stack of scopes currently, uh, in scope. Each\nelement in the stack is a Map representing a single block scope. Keys, as in\nEnvironment, are variable names. The values are Booleans, for a reason I'll\nexplain soon.\n\nThe scope stack is only used for local block scopes. Variables declared at the\ntop level in the global scope are not tracked by the resolver since they are\nmore dynamic in Lox. When resolving a variable, if we can't find it in the stack\nof local scopes, we assume it must be global.\n\nSince scopes are stored in an explicit stack, exiting one is straightforward.\n\n^code end-scope\n\nNow we can push and pop a stack of empty scopes. Let's put some things in them.\n\n### Resolving variable declarations\n\nResolving a variable declaration adds a new entry to the current innermost\nscope's map. That seems simple, but there's a little dance we need to do.\n\n^code visit-var-stmt\n\nWe split binding into two steps, declaring then defining, in order to handle\nfunny edge cases like this:\n\n```lox\nvar a = \"outer\";\n{\n var a = a;\n}\n```\n\nWhat happens when the initializer for a local variable refers to a variable with\nthe same name as the variable being declared? We have a few options:\n\n1. **Run the initializer, then put the new variable in scope.** Here, the new\n local `a` would be initialized with \"outer\", the value of the *global* one.\n In other words, the previous declaration would desugar to:\n\n ```lox\n var temp = a; // Run the initializer.\n var a; // Declare the variable.\n a = temp; // Initialize it.\n ```\n\n2. **Put the new variable in scope, then run the initializer.** This means you\n could observe a variable before it's initialized, so we would need to figure\n out what value it would have then. Probably `nil`. That means the new local\n `a` would be re-initialized to its own implicitly initialized value, `nil`.\n Now the desugaring would look like:\n\n ```lox\n var a; // Define the variable.\n a = a; // Run the initializer.\n ```\n\n3. **Make it an error to reference a variable in its initializer.** Have the\n interpreter fail either at compile time or runtime if an initializer\n mentions the variable being initialized.\n\nDo either of those first two options look like something a user actually\n*wants*? Shadowing is rare and often an error, so initializing a shadowing\nvariable based on the value of the shadowed one seems unlikely to be deliberate.\n\nThe second option is even less useful. The new variable will *always* have the\nvalue `nil`. There is never any point in mentioning it by name. You could use an\nexplicit `nil` instead.\n\nSince the first two options are likely to mask user errors, we'll take the\nthird. Further, we'll make it a compile error instead of a runtime one. That\nway, the user is alerted to the problem before any code is run.\n\nIn order to do that, as we visit expressions, we need to know if we're inside\nthe initializer for some variable. We do that by splitting binding into two\nsteps. The first is **declaring** it.\n\n^code declare\n\nDeclaration adds the variable to the innermost scope so that it shadows any\nouter one and so that we know the variable exists. We mark it as \"not ready yet\"\nby binding its name to `false` in the scope map. The value associated with a key\nin the scope map represents whether or not we have finished resolving that\nvariable's initializer.\n\nAfter declaring the variable, we resolve its initializer expression in that same\nscope where the new variable now exists but is unavailable. Once the initializer\nexpression is done, the variable is ready for prime time. We do that by\n**defining** it.\n\n^code define\n\nWe set the variable's value in the scope map to `true` to mark it as fully\ninitialized and available for use. It's alive! \n\n### Resolving variable expressions\n\nVariable declarations -- and function declarations, which we'll get to -- write\nto the scope maps. Those maps are read when we resolve variable expressions.\n\n^code visit-variable-expr\n\nFirst, we check to see if the variable is being accessed inside its own\ninitializer. This is where the values in the scope map come into play. If the\nvariable exists in the current scope but its value is `false`, that means we\nhave declared it but not yet defined it. We report that error.\n\nAfter that check, we actually resolve the variable itself using this helper:\n\n^code resolve-local\n\nThis looks, for good reason, a lot like the code in Environment for evaluating a\nvariable. We start at the innermost scope and work outwards, looking in each map\nfor a matching name. If we find the variable, we resolve it, passing in the\nnumber of scopes between the current innermost scope and the scope where the\nvariable was found. So, if the variable was found in the current scope, we\npass in 0. If it's in the immediately enclosing scope, 1. You get the idea.\n\nIf we walk through all of the block scopes and never find the variable, we leave\nit unresolved and assume it's global. We'll get to the implementation of that\n`resolve()` method a little later. For now, let's keep on cranking through the\nother syntax nodes.\n\n### Resolving assignment expressions\n\nThe other expression that references a variable is assignment. Resolving one\nlooks like this:\n\n^code visit-assign-expr\n\nFirst, we resolve the expression for the assigned value in case it also contains\nreferences to other variables. Then we use our existing `resolveLocal()` method\nto resolve the variable that's being assigned to.\n\n### Resolving function declarations\n\nFinally, functions. Functions both bind names and introduce a scope. The name of\nthe function itself is bound in the surrounding scope where the function is\ndeclared. When we step into the function's body, we also bind its parameters\ninto that inner function scope.\n\n^code visit-function-stmt\n\nSimilar to `visitVariableStmt()`, we declare and define the name of the function\nin the current scope. Unlike variables, though, we define the name eagerly,\nbefore resolving the function's body. This lets a function recursively refer to\nitself inside its own body.\n\nThen we resolve the function's body using this:\n\n^code resolve-function\n\nIt's a separate method since we will also use it for resolving Lox methods when\nwe add classes later. It creates a new scope for the body and then binds\nvariables for each of the function's parameters.\n\nOnce that's ready, it resolves the function body in that scope. This is\ndifferent from how the interpreter handles function declarations. At *runtime*,\ndeclaring a function doesn't do anything with the function's body. The body\ndoesn't get touched until later when the function is called. In a *static*\nanalysis, we immediately traverse into the body right then and there.\n\n### Resolving the other syntax tree nodes\n\nThat covers the interesting corners of the grammars. We handle every place where\na variable is declared, read, or written, and every place where a scope is\ncreated or destroyed. Even though they aren't affected by variable resolution,\nwe also need visit methods for all of the other syntax tree nodes in order to\nrecurse into their subtrees. Sorry this bit is\nboring, but bear with me. We'll go kind of \"top down\" and start with statements.\n\n\n\nAn expression statement contains a single expression to traverse.\n\n^code visit-expression-stmt\n\nAn if statement has an expression for its condition and one or two statements\nfor the branches.\n\n^code visit-if-stmt\n\nHere, we see how resolution is different from interpretation. When we resolve an\n`if` statement, there is no control flow. We resolve the condition and *both*\nbranches. Where a dynamic execution steps only into the branch that *is* run, a\nstatic analysis is conservative -- it analyzes any branch that *could* be run.\nSince either one could be reached at runtime, we resolve both.\n\nLike expression statements, a `print` statement contains a single subexpression.\n\n^code visit-print-stmt\n\nSame deal for return.\n\n^code visit-return-stmt\n\nAs in `if` statements, with a `while` statement, we resolve its condition and\nresolve the body exactly once.\n\n^code visit-while-stmt\n\nThat covers all the statements. On to expressions...\n\nOur old friend the binary expression. We traverse into and resolve both\noperands.\n\n^code visit-binary-expr\n\nCalls are similar -- we walk the argument list and resolve them all. The thing\nbeing called is also an expression (usually a variable expression), so that gets\nresolved too.\n\n^code visit-call-expr\n\nParentheses are easy.\n\n^code visit-grouping-expr\n\nLiterals are easiest of all.\n\n^code visit-literal-expr\n\nA literal expression doesn't mention any variables and doesn't contain any\nsubexpressions so there is no work to do.\n\nSince a static analysis does no control flow or short-circuiting, logical\nexpressions are exactly the same as other binary operators.\n\n^code visit-logical-expr\n\nAnd, finally, the last node. We resolve its one operand.\n\n^code visit-unary-expr\n\nWith all of these visit methods, the Java compiler should be satisfied that\nResolver fully implements Stmt.Visitor and Expr.Visitor. Now is a good time to\ntake a break, have a snack, maybe a little nap.\n\n## Interpreting Resolved Variables\n\nLet's see what our resolver is good for. Each time it visits a variable, it\ntells the interpreter how many scopes there are between the current scope and\nthe scope where the variable is defined. At runtime, this corresponds exactly to\nthe number of *environments* between the current one and the enclosing one where\nthe interpreter can find the variable's value. The resolver hands that number to\nthe interpreter by calling this:\n\n^code resolve\n\nWe want to store the resolution information somewhere so we can use it when the\nvariable or assignment expression is later executed, but where? One obvious\nplace is right in the syntax tree node itself. That's a fine approach, and\nthat's where many compilers store the results of analyses like this.\n\nWe could do that, but it would require mucking around with our syntax tree\ngenerator. Instead, we'll take another common approach and store it off to the\nside in a map that associates each syntax tree node\nwith its resolved data.\n\n\n\nInteractive tools like IDEs often incrementally reparse and re-resolve parts of\nthe user's program. It may be hard to find all of the bits of state that need\nrecalculating when they're hiding in the foliage of the syntax tree. A benefit\nof storing this data outside of the nodes is that it makes it easy to *discard*\nit -- simply clear the map.\n\n^code locals-field (1 before, 2 after)\n\nYou might think we'd need some sort of nested tree structure to avoid getting\nconfused when there are multiple expressions that reference the same variable,\nbut each expression node is its own Java object with its own unique identity. A\nsingle monolithic map doesn't have any trouble keeping them separated.\n\nAs usual, using a collection requires us to import a couple of names.\n\n^code import-hash-map (1 before, 1 after)\n\nAnd:\n\n^code import-map (1 before, 2 after)\n\n### Accessing a resolved variable\n\nOur interpreter now has access to each variable's resolved location. Finally, we\nget to make use of that. We replace the visit method for variable expressions\nwith this:\n\n^code call-look-up-variable (1 before, 1 after)\n\nThat delegates to:\n\n^code look-up-variable\n\nThere are a couple of things going on here. First, we look up the resolved\ndistance in the map. Remember that we resolved only *local* variables. Globals\nare treated specially and don't end up in the map (hence the name `locals`). So,\nif we don't find a distance in the map, it must be global. In that case, we\nlook it up, dynamically, directly in the global environment. That throws a\nruntime error if the variable isn't defined.\n\nIf we *do* get a distance, we have a local variable, and we get to take\nadvantage of the results of our static analysis. Instead of calling `get()`, we\ncall this new method on Environment:\n\n^code get-at\n\nThe old `get()` method dynamically walks the chain of enclosing environments,\nscouring each one to see if the variable might be hiding in there somewhere. But\nnow we know exactly which environment in the chain will have the variable. We\nreach it using this helper method:\n\n^code ancestor\n\nThis walks a fixed number of hops up the parent chain and returns the\nenvironment there. Once we have that, `getAt()` simply returns the value of the\nvariable in that environment's map. It doesn't even have to check to see if the\nvariable is there -- we know it will be because the resolver already found it\nbefore.\n\n\n\n### Assigning to a resolved variable\n\nWe can also use a variable by assigning to it. The changes to visiting an\nassignment expression are similar.\n\n^code resolved-assign (2 before, 1 after)\n\nAgain, we look up the variable's scope distance. If not found, we assume it's\nglobal and handle it the same way as before. Otherwise, we call this new method:\n\n^code assign-at\n\nAs `getAt()` is to `get()`, `assignAt()` is to `assign()`. It walks a fixed\nnumber of environments, and then stuffs the new value in that map.\n\nThose are the only changes to Interpreter. This is why I chose a representation\nfor our resolved data that was minimally invasive. All of the rest of the nodes\ncontinue working as they did before. Even the code for modifying environments is\nunchanged.\n\n### Running the resolver\n\nWe do need to actually *run* the resolver, though. We insert the new pass after\nthe parser does its magic.\n\n^code create-resolver (3 before, 1 after)\n\nWe don't run the resolver if there are any parse errors. If the code has a\nsyntax error, it's never going to run, so there's little value in resolving it.\nIf the syntax is clean, we tell the resolver to do its thing. The resolver has a\nreference to the interpreter and pokes the resolution data directly into it as\nit walks over variables. When the interpreter runs next, it has everything it\nneeds.\n\nAt least, that's true if the resolver *succeeds*. But what about errors during\nresolution?\n\n## Resolution Errors\n\nSince we are doing a semantic analysis pass, we have an opportunity to make\nLox's semantics more precise, and to help users catch bugs early before running\ntheir code. Take a look at this bad boy:\n\n```lox\nfun bad() {\n var a = \"first\";\n var a = \"second\";\n}\n```\n\nWe do allow declaring multiple variables with the same name in the *global*\nscope, but doing so in a local scope is probably a mistake. If they knew the\nvariable already existed, they would have assigned to it instead of using `var`.\nAnd if they *didn't* know it existed, they probably didn't intend to overwrite\nthe previous one.\n\nWe can detect this mistake statically while resolving.\n\n^code duplicate-variable (1 before, 1 after)\n\nWhen we declare a variable in a local scope, we already know the names of every\nvariable previously declared in that same scope. If we see a collision, we\nreport an error.\n\n### Invalid return errors\n\nHere's another nasty little script:\n\n```lox\nreturn \"at top level\";\n```\n\nThis executes a `return` statement, but it's not even inside a function at all.\nIt's top-level code. I don't know what the user *thinks* is going to happen, but\nI don't think we want Lox to allow this.\n\nWe can extend the resolver to detect this statically. Much like we track scopes\nas we walk the tree, we can track whether or not the code we are currently\nvisiting is inside a function declaration.\n\n^code function-type-field (1 before, 2 after)\n\nInstead of a bare Boolean, we use this funny enum:\n\n^code function-type\n\nIt seems kind of dumb now, but we'll add a couple more cases to it later and\nthen it will make more sense. When we resolve a function declaration, we pass\nthat in.\n\n^code pass-function-type (2 before, 1 after)\n\nOver in `resolveFunction()`, we take that parameter and store it in the field\nbefore resolving the body.\n\n^code set-current-function (1 after)\n\nWe stash the previous value of the field in a local variable first. Remember,\nLox has local functions, so you can nest function declarations arbitrarily\ndeeply. We need to track not just that we're in a function, but *how many* we're\nin.\n\nWe could use an explicit stack of FunctionType values for that, but instead\nwe'll piggyback on the JVM. We store the previous value in a local on the Java\nstack. When we're done resolving the function body, we restore the field to that\nvalue.\n\n^code restore-current-function (1 before, 1 after)\n\nNow that we can always tell whether or not we're inside a function declaration,\nwe check that when resolving a `return` statement.\n\n^code return-from-top (1 before, 1 after)\n\nNeat, right?\n\nThere's one more piece. Back in the main Lox class that stitches everything\ntogether, we are careful to not run the interpreter if any parse errors are\nencountered. That check runs *before* the resolver so that we don't try to\nresolve syntactically invalid code.\n\nBut we also need to skip the interpreter if there are resolution errors, so we\nadd *another* check.\n\n^code resolution-error (1 before, 2 after)\n\nYou could imagine doing lots of other analysis in here. For example, if we added\n`break` statements to Lox, we would probably want to ensure they are only used\ninside loops.\n\nWe could go farther and report warnings for code that isn't necessarily *wrong*\nbut probably isn't useful. For example, many IDEs will warn if you have\nunreachable code after a `return` statement, or a local variable whose value is\nnever read. All of that would be pretty easy to add to our static visiting pass,\nor as separate passes.\n\n\n\nBut, for now, we'll stick with that limited amount of analysis. The important\npart is that we fixed that one weird annoying edge case bug, though it might be\nsurprising that it took this much work to do it.\n\n
\n\n## Challenges\n\n1. Why is it safe to eagerly define the variable bound to a function's name\n when other variables must wait until after they are initialized before they\n can be used?\n\n1. How do other languages you know handle local variables that refer to the\n same name in their initializer, like:\n\n ```lox\n var a = \"outer\";\n {\n var a = a;\n }\n ```\n\n Is it a runtime error? Compile error? Allowed? Do they treat global\n variables differently? Do you agree with their choices? Justify your answer.\n\n1. Extend the resolver to report an error if a local variable is never used.\n\n1. Our resolver calculates *which* environment the variable is found in, but\n it's still looked up by name in that map. A more efficient environment\n representation would store local variables in an array and look them up by\n index.\n\n Extend the resolver to associate a unique index for each local variable\n declared in a scope. When resolving a variable access, look up both the\n scope the variable is in and its index and store that. In the interpreter,\n use that to quickly access a variable by its index instead of using a map.\n\n
\n" }, { "path": "book/scanning-on-demand.md", "content": "> Literature is idiosyncratic arrangements in horizontal lines in only\n> twenty-six phonetic symbols, ten Arabic numbers, and about eight punctuation\n> marks.\n>\n> Kurt Vonnegut, Like Shaking Hands With God: A Conversation about Writing\n\nOur second interpreter, clox, has three phases -- scanner, compiler, and virtual\nmachine. A data structure joins each pair of phases. Tokens flow from scanner to\ncompiler, and chunks of bytecode from compiler to VM. We began our\nimplementation near the end with [chunks][] and the [VM][]. Now, we're going to\nhop back to the beginning and build a scanner that makes tokens. In the\n[next chapter][], we'll tie the two ends together with our bytecode compiler.\n\n[chunks]: chunks-of-bytecode.html\n[vm]: a-virtual-machine.html\n[next chapter]: compiling-expressions.html\n\n\"Source\n\nI'll admit, this is not the most exciting chapter in the book. With two\nimplementations of the same language, there's bound to be some redundancy. I did\nsneak in a few interesting differences compared to jlox's scanner. Read on to\nsee what they are.\n\n## Spinning Up the Interpreter\n\nNow that we're building the front end, we can get clox running like a real\ninterpreter. No more hand-authored chunks of bytecode. It's time for a REPL and\nscript loading. Tear out most of the code in `main()` and replace it with:\n\n^code args (3 before, 2 after)\n\nIf you pass no arguments to the executable, you are\ndropped into the REPL. A single command line argument is understood to be the\npath to a script to run.\n\n\n\nWe'll need a few system headers, so let's get them all out of the way.\n\n^code main-includes (1 after)\n\nNext, we get the REPL up and REPL-ing.\n\n^code repl (1 before)\n\nA quality REPL handles input that spans multiple lines gracefully and doesn't\nhave a hardcoded line length limit. This REPL here is a little more, ahem,\naustere, but it's fine for our purposes.\n\nThe real work happens in `interpret()`. We'll get to that soon, but first let's\ntake care of loading scripts.\n\n^code run-file\n\nWe read the file and execute the resulting string of Lox source code. Then,\nbased on the result of that, we set the exit code appropriately because we're\nscrupulous tool builders and care about little details like that.\n\nWe also need to free the source code string because `readFile()` dynamically\nallocates it and passes ownership to its caller. That function looks like this:\n\n\n\n^code read-file\n\nLike a lot of C code, it takes more effort than it seems like it should,\nespecially for a language expressly designed for operating systems. The\ndifficult part is that we want to allocate a big enough string to read the whole\nfile, but we don't know how big the file is until we've read it.\n\nThe code here is the classic trick to solve that. We open the file, but before\nreading it, we seek to the very end using `fseek()`. Then we call `ftell()`\nwhich tells us how many bytes we are from the start of the file. Since we seeked\n(sought?) to the end, that's the size. We rewind back to the beginning, allocate\na string of that size, and read the whole file in a\nsingle batch.\n\n\n\nSo we're done, right? Not quite. These function calls, like most calls in the C\nstandard library, can fail. If this were Java, the failures would be thrown as\nexceptions and automatically unwind the stack so we wouldn't *really* need to\nhandle them. In C, if we don't check for them, they silently get ignored.\n\nThis isn't really a book on good C programming practice, but I hate to encourage\nbad style, so let's go ahead and handle the errors. It's good for us, like\neating our vegetables or flossing.\n\nFortunately, we don't need to do anything particularly clever if a failure\noccurs. If we can't correctly read the user's script, all we can really do is\ntell the user and exit the interpreter gracefully. First of all, we might fail\nto open the file.\n\n^code no-file (1 before, 2 after)\n\nThis can happen if the file doesn't exist or the user doesn't have access to it.\nIt's pretty common -- people mistype paths all the time.\n\nThis failure is much rarer:\n\n^code no-buffer (1 before, 1 after)\n\nIf we can't even allocate enough memory to read the Lox script, the user's\nprobably got bigger problems to worry about, but we should do our best to at\nleast let them know.\n\nFinally, the read itself may fail.\n\n^code no-read (1 before, 1 after)\n\nThis is also unlikely. Actually, the calls to\n`fseek()`, `ftell()`, and `rewind()` could theoretically fail too, but let's not\ngo too far off in the weeds, shall we?\n\n\n\n### Opening the compilation pipeline\n\nWe've got ourselves a string of Lox source code, so now we're ready to set up a\npipeline to scan, compile, and execute it. It's driven by `interpret()`. Right\nnow, that function runs our old hardcoded test chunk. Let's change it to\nsomething closer to its final incarnation.\n\n^code vm-interpret-h (1 before, 1 after)\n\nWhere before we passed in a Chunk, now we pass in the string of source code.\nHere's the new implementation:\n\n^code vm-interpret-c (1 after)\n\nWe won't build the actual *compiler* yet in this chapter, but we can start\nlaying out its structure. It lives in a new module.\n\n^code vm-include-compiler (1 before, 1 after)\n\nFor now, the one function in it is declared like so:\n\n^code compiler-h\n\nThat signature will change, but it gets us going.\n\nThe first phase of compilation is scanning -- the thing we're doing in this\nchapter -- so right now all the compiler does is set that up.\n\n^code compiler-c\n\nThis will also grow in later chapters, naturally.\n\n### The scanner scans\n\nThere are still a few more feet of scaffolding to stand up before we can start\nwriting useful code. First, a new header:\n\n^code scanner-h\n\nAnd its corresponding implementation:\n\n^code scanner-c\n\nAs our scanner chews through the user's source code, it tracks how far it's\ngone. Like we did with the VM, we wrap that state in a struct and then create a\nsingle top-level module variable of that type so we don't have to pass it around\nall of the various functions.\n\nThere are surprisingly few fields. The `start` pointer marks the beginning of\nthe current lexeme being scanned, and `current` points to the current character\nbeing looked at.\n\n\n\n\"The\n\n\n\nWe have a `line` field to track what line the current lexeme is on for error\nreporting. That's it! We don't even keep a pointer to the beginning of the\nsource code string. The scanner works its way through the code once and is done\nafter that.\n\nSince we have some state, we should initialize it.\n\n^code init-scanner\n\nWe start at the very first character on the very first line, like a runner\ncrouched at the starting line.\n\n## A Token at a Time\n\nIn jlox, when the starting gun went off, the scanner raced ahead and eagerly\nscanned the whole program, returning a list of tokens. This would be a challenge\nin clox. We'd need some sort of growable array or list to store the tokens in.\nWe'd need to manage allocating and freeing the tokens, and the collection\nitself. That's a lot of code, and a lot of memory churn.\n\nAt any point in time, the compiler needs only one or two tokens -- remember our\ngrammar requires only a single token of lookahead -- so we don't need to keep\nthem *all* around at the same time. Instead, the simplest solution is to not\nscan a token until the compiler needs one. When the scanner provides one, it\nreturns the token by value. It doesn't need to dynamically allocate anything --\nit can just pass tokens around on the C stack.\n\nUnfortunately, we don't have a compiler yet that can ask the scanner for tokens,\nso the scanner will just sit there doing nothing. To kick it into action, we'll\nwrite some temporary code to drive it.\n\n^code dump-tokens (1 before, 1 after)\n\n\n\nThis loops indefinitely. Each turn through the loop, it scans one token and\nprints it. When it reaches a special \"end of file\" token or an error, it stops.\nFor example, if we run the interpreter on this program:\n\n```lox\nprint 1 + 2;\n```\n\nIt prints out:\n\n```text\n 1 31 'print'\n | 21 '1'\n | 7 '+'\n | 21 '2'\n | 8 ';'\n 2 39 ''\n```\n\nThe first column is the line number, the second is the numeric value of the\ntoken type, and then finally the lexeme. That last\nempty lexeme on line 2 is the EOF token.\n\n\n\nThe goal for the rest of the chapter is to make that blob of code work by\nimplementing this key function:\n\n^code scan-token-h (1 before, 2 after)\n\nEach call scans and returns the next token in the source code. A token looks\nlike this:\n\n^code token-struct (1 before, 2 after)\n\nIt's pretty similar to jlox's Token class. We have an enum identifying what type\nof token it is -- number, identifier, `+` operator, etc. The enum is virtually\nidentical to the one in jlox, so let's just hammer out the whole thing.\n\n^code token-type (2 before, 2 after)\n\nAside from prefixing all the names with `TOKEN_` (since C tosses enum names in\nthe top-level namespace) the only difference is that extra `TOKEN_ERROR` type.\nWhat's that about?\n\nThere are only a couple of errors that get detected during scanning:\nunterminated strings and unrecognized characters. In jlox, the scanner reports\nthose itself. In clox, the scanner produces a synthetic \"error\" token for that\nerror and passes it over to the compiler. This way, the compiler knows an error\noccurred and can kick off error recovery before reporting it.\n\nThe novel part in clox's Token type is how it represents the lexeme. In jlox,\neach Token stored the lexeme as its own separate little Java string. If we did\nthat for clox, we'd have to figure out how to manage the memory for those\nstrings. That's especially hard since we pass tokens by value\n-- multiple tokens could point to the same lexeme string. Ownership gets weird.\n\nInstead, we use the original source string as our character store. We represent\na lexeme by a pointer to its first character and the number of characters it\ncontains. This means we don't need to worry about managing memory for lexemes at\nall and we can freely copy tokens around. As long as the main source code string\noutlives all of the tokens, everything works fine.\n\n\n\n### Scanning tokens\n\nWe're ready to scan some tokens. We'll work our way up to the complete\nimplementation, starting with this:\n\n^code scan-token\n\nSince each call to this function scans a complete token, we know we are at the\nbeginning of a new token when we enter the function. Thus, we set\n`scanner.start` to point to the current character so we remember where the\nlexeme we're about to scan starts.\n\nThen we check to see if we've reached the end of the source code. If so, we\nreturn an EOF token and stop. This is a sentinel value that signals to the\ncompiler to stop asking for more tokens.\n\nIf we aren't at the end, we do some... stuff... to scan the next token. But we\nhaven't written that code yet. We'll get to that soon. If that code doesn't\nsuccessfully scan and return a token, then we reach the end of the function.\nThat must mean we're at a character that the scanner can't recognize, so we\nreturn an error token for that.\n\nThis function relies on a couple of helpers, most of which are familiar from\njlox. First up:\n\n^code is-at-end\n\nWe require the source string to be a good null-terminated C string. If the\ncurrent character is the null byte, then we've reached the end.\n\nTo create a token, we have this constructor-like function:\n\n^code make-token\n\nIt uses the scanner's `start` and `current` pointers to capture the token's\nlexeme. It sets a couple of other obvious fields then returns the token. It has\na sister function for returning error tokens.\n\n^code error-token\n\n\n\n\n\nThe only difference is that the \"lexeme\" points to the error message string\ninstead of pointing into the user's source code. Again, we need to ensure that\nthe error message sticks around long enough for the compiler to read it. In\npractice, we only ever call this function with C string literals. Those are\nconstant and eternal, so we're fine.\n\nWhat we have now is basically a working scanner for a language with an empty\nlexical grammar. Since the grammar has no productions, every character is an\nerror. That's not exactly a fun language to program in, so let's fill in the\nrules.\n\n## A Lexical Grammar for Lox\n\nThe simplest tokens are only a single character. We recognize those like so:\n\n^code scan-char (1 before, 2 after)\n\nWe read the next character from the source code, and then do a straightforward\nswitch to see if it matches any of Lox's one-character lexemes. To read the next\ncharacter, we use a new helper which consumes the current character and returns\nit.\n\n^code advance\n\nNext up are the two-character punctuation tokens like `!=` and `>=`. Each of\nthese also has a corresponding single-character token. That means that when we\nsee a character like `!`, we don't know if we're in a `!` token or a `!=` until\nwe look at the next character too. We handle those like so:\n\n^code two-char (1 before, 1 after)\n\nAfter consuming the first character, we look for an `=`. If found, we consume it\nand return the corresponding two-character token. Otherwise, we leave the\ncurrent character alone (so it can be part of the *next* token) and return the\nappropriate one-character token.\n\nThat logic for conditionally consuming the second character lives here:\n\n^code match\n\nIf the current character is the desired one, we advance and return `true`.\nOtherwise, we return `false` to indicate it wasn't matched.\n\nNow our scanner supports all of the punctuation-like tokens. Before we get to\nthe longer ones, let's take a little side trip to handle characters that aren't\npart of a token at all.\n\n### Whitespace\n\nOur scanner needs to handle spaces, tabs, and newlines, but those characters\ndon't become part of any token's lexeme. We could check for those inside the\nmain character switch in `scanToken()` but it gets a little tricky to ensure\nthat the function still correctly finds the next token *after* the whitespace\nwhen you call it. We'd have to wrap the whole body of the function in a loop or\nsomething.\n\nInstead, before starting the token, we shunt off to a separate function.\n\n^code call-skip-whitespace (1 before, 1 after)\n\nThis advances the scanner past any leading whitespace. After this call returns,\nwe know the very next character is a meaningful one (or we're at the end of the\nsource code).\n\n^code skip-whitespace\n\nIt's sort of a separate mini-scanner. It loops, consuming every whitespace\ncharacter it encounters. We need to be careful that it does *not* consume any\n*non*-whitespace characters. To support that, we use this:\n\n^code peek\n\nThis simply returns the current character, but doesn't consume it. The previous\ncode handles all the whitespace characters except for newlines.\n\n^code newline (1 before, 2 after)\n\nWhen we consume one of those, we also bump the current line number.\n\n### Comments\n\nComments aren't technically \"whitespace\", if you want to get all precise with\nyour terminology, but as far as Lox is concerned, they may as well be, so we\nskip those too.\n\n^code comment (1 before, 2 after)\n\nComments start with `//` in Lox, so as with `!=` and friends, we need a second\ncharacter of lookahead. However, with `!=`, we still wanted to consume the `!`\neven if the `=` wasn't found. Comments are different. If we don't find a second\n`/`, then `skipWhitespace()` needs to not consume the *first* slash either.\n\nTo handle that, we add:\n\n^code peek-next\n\nThis is like `peek()` but for one character past the current one. If the current\ncharacter and the next one are both `/`, we consume them and then any other\ncharacters until the next newline or the end of the source code.\n\nWe use `peek()` to check for the newline but not consume it. That way, the\nnewline will be the current character on the next turn of the outer loop in\n`skipWhitespace()` and we'll recognize it and increment `scanner.line`.\n\n### Literal tokens\n\nNumber and string tokens are special because they have a runtime value\nassociated with them. We'll start with strings because they are easy to\nrecognize -- they always begin with a double quote.\n\n^code scan-string (1 before, 1 after)\n\nThat calls a new function.\n\n^code string\n\nSimilar to jlox, we consume characters until we reach the closing quote. We also\ntrack newlines inside the string literal. (Lox supports multi-line strings.)\nAnd, as ever, we gracefully handle running out of source code before we find the\nend quote.\n\nThe main change here in clox is something that's *not* present. Again, it\nrelates to memory management. In jlox, the Token class had a field of type\nObject to store the runtime value converted from the literal token's lexeme.\n\nImplementing that in C would require a lot of work. We'd need some sort of union\nand type tag to tell whether the token contains a string or double value. If\nit's a string, we'd need to manage the memory for the string's character array\nsomehow.\n\nInstead of adding that complexity to the scanner, we defer converting the literal lexeme to a runtime value until\nlater. In clox, tokens only store the lexeme -- the character sequence exactly\nas it appears in the user's source code. Later in the compiler, we'll convert\nthat lexeme to a runtime value right when we are ready to store it in the\nchunk's constant table.\n\n\n\nNext up, numbers. Instead of adding a switch case for each of the ten digits\nthat can start a number, we handle them here:\n\n^code scan-number (1 before, 2 after)\n\nThat uses this obvious utility function:\n\n^code is-digit\n\nWe finish scanning the number using this:\n\n^code number\n\nIt's virtually identical to jlox's version except, again, we don't convert the\nlexeme to a double yet.\n\n## Identifiers and Keywords\n\nThe last batch of tokens are identifiers, both user-defined and reserved. This\nsection should be fun -- the way we recognize keywords in clox is quite\ndifferent from how we did it in jlox, and touches on some important data\nstructures.\n\nFirst, though, we have to scan the lexeme. Names start with a letter or\nunderscore.\n\n^code scan-identifier (1 before, 1 after)\n\nWe recognize those using this:\n\n^code is-alpha\n\nOnce we've found an identifier, we scan the rest of it here:\n\n^code identifier\n\nAfter the first letter, we allow digits too, and we keep consuming alphanumerics\nuntil we run out of them. Then we produce a token with the proper type.\nDetermining that \"proper\" type is the unique part of this chapter.\n\n^code identifier-type\n\nOkay, I guess that's not very exciting yet. That's what it looks like if we\nhave no reserved words at all. How should we go about recognizing keywords? In\njlox, we stuffed them all in a Java Map and looked them up by name. We don't\nhave any sort of hash table structure in clox, at least not yet.\n\nA hash table would be overkill anyway. To look up a string in a hash table, we need to walk the string to calculate its hash code,\nfind the corresponding bucket in the hash table, and then do a\ncharacter-by-character equality comparison on any string it happens to find\nthere.\n\n\n\nLet's say we've scanned the identifier \"gorgonzola\". How much work *should* we\nneed to do to tell if that's a reserved word? Well, no Lox keyword starts with\n\"g\", so looking at the first character is enough to definitively answer no.\nThat's a lot simpler than a hash table lookup.\n\nWhat about \"cardigan\"? We do have a keyword in Lox that starts with \"c\":\n\"class\". But the second character in \"cardigan\", \"a\", rules that out. What about\n\"forest\"? Since \"for\" is a keyword, we have to go farther in the string before\nwe can establish that we don't have a reserved word. But, in most cases, only a\ncharacter or two is enough to tell we've got a user-defined name on our hands.\nWe should be able to recognize that and fail fast.\n\nHere's a visual representation of that branching character-inspection logic:\n\n\n\n\"A\n\n\n\nWe start at the root node. If there is a child node whose letter matches the\nfirst character in the lexeme, we move to that node. Then repeat for the next\nletter in the lexeme and so on. If at any point the next letter in the lexeme\ndoesn't match a child node, then the identifier must not be a keyword and we\nstop. If we reach a double-lined box, and we're at the last character of the\nlexeme, then we found a keyword.\n\n### Tries and state machines\n\nThis tree diagram is an example of a thing called a [**trie**][trie]. A trie stores a set of strings. Most other\ndata structures for storing strings contain the raw character arrays and then\nwrap them inside some larger construct that helps you search faster. A trie is\ndifferent. Nowhere in the trie will you find a whole string.\n\n[trie]: https://en.wikipedia.org/wiki/Trie\n\n\n\nInstead, each string the trie \"contains\" is represented as a *path* through the\ntree of character nodes, as in our traversal above. Nodes that match the last\ncharacter in a string have a special marker -- the double lined boxes in the\nillustration. That way, if your trie contains, say, \"banquet\" and \"ban\", you are\nable to tell that it does *not* contain \"banque\" -- the \"e\" node won't have that\nmarker, while the \"n\" and \"t\" nodes will.\n\nTries are a special case of an even more fundamental data structure: a\n[**deterministic finite automaton**][dfa] (**DFA**). You might also know these\nby other names: **finite state machine**, or just **state machine**. State\nmachines are rad. They end up useful in everything from [game\nprogramming][state] to implementing networking protocols.\n\n[dfa]: https://en.wikipedia.org/wiki/Deterministic_finite_automaton\n[state]: http://gameprogrammingpatterns.com/state.html\n\nIn a DFA, you have a set of *states* with *transitions* between them, forming a\ngraph. At any point in time, the machine is \"in\" exactly one state. It gets to\nother states by following transitions. When you use a DFA for lexical analysis,\neach transition is a character that gets matched from the string. Each state\nrepresents a set of allowed characters.\n\nOur keyword tree is exactly a DFA that recognizes Lox keywords. But DFAs are\nmore powerful than simple trees because they can be arbitrary *graphs*.\nTransitions can form cycles between states. That lets you recognize arbitrarily\nlong strings. For example, here's a DFA that recognizes number literals:\n\n\n\n\"A\n\n\n\nI've collapsed the nodes for the ten digits together to keep it more readable,\nbut the basic process works the same -- you work through the path, entering\nnodes whenever you consume a corresponding character in the lexeme. If we were\nso inclined, we could construct one big giant DFA that does *all* of the lexical\nanalysis for Lox, a single state machine that recognizes and spits out all of\nthe tokens we need.\n\nHowever, crafting that mega-DFA by hand would be\nchallenging. That's why [Lex][] was created. You give it a simple textual\ndescription of your lexical grammar -- a bunch of regular expressions -- and it\nautomatically generates a DFA for you and produces a pile of C code that\nimplements it.\n\n[lex]: https://en.wikipedia.org/wiki/Lex_(software)\n\n\n\nWe won't go down that road. We already have a perfectly serviceable hand-rolled\nscanner. We just need a tiny trie for recognizing keywords. How should we map\nthat to code?\n\nThe absolute simplest solution is to use a switch\nstatement for each node with cases for each branch. We'll start with the root\nnode and handle the easy keywords.\n\n\n\n^code keywords (1 before, 1 after)\n\nThese are the initial letters that correspond to a single keyword. If we see an\n\"s\", the only keyword the identifier could possibly be is `super`. It might not\nbe, though, so we still need to check the rest of the letters too. In the tree\ndiagram, this is basically that straight path hanging off the \"s\".\n\nWe won't roll a switch for each of those nodes. Instead, we have a utility\nfunction that tests the rest of a potential keyword's lexeme.\n\n^code check-keyword\n\nWe use this for all of the unbranching paths in the tree. Once we've found a\nprefix that could only be one possible reserved word, we need to verify two\nthings. The lexeme must be exactly as long as the keyword. If the first letter\nis \"s\", the lexeme could still be \"sup\" or \"superb\". And the remaining\ncharacters must match exactly -- \"supar\" isn't good enough.\n\nIf we do have the right number of characters, and they're the ones we want, then\nit's a keyword, and we return the associated token type. Otherwise, it must be a\nnormal identifier.\n\nWe have a couple of keywords where the tree branches again after the first\nletter. If the lexeme starts with \"f\", it could be `false`, `for`, or `fun`. So\nwe add another switch for the branches coming off the \"f\" node.\n\n^code keyword-f (1 before, 1 after)\n\nBefore we switch, we need to check that there even *is* a second letter. \"f\" by\nitself is a valid identifier too, after all. The other letter that branches is\n\"t\".\n\n^code keyword-t (1 before, 1 after)\n\nThat's it. A couple of nested `switch` statements. Not only is this code short, but it's very, very fast. It does the minimum amount\nof work required to detect a keyword, and bails out as soon as it can tell the\nidentifier will not be a reserved one.\n\nAnd with that, our scanner is complete.\n\n\n\n
\n\n## Challenges\n\n1. Many newer languages support [**string interpolation**][interp]. Inside a\n string literal, you have some sort of special delimiters -- most commonly\n `${` at the beginning and `}` at the end. Between those delimiters, any\n expression can appear. When the string literal is executed, the inner\n expression is evaluated, converted to a string, and then merged with the\n surrounding string literal.\n\n For example, if Lox supported string interpolation, then this...\n\n ```lox\n var drink = \"Tea\";\n var steep = 4;\n var cool = 2;\n print \"${drink} will be ready in ${steep + cool} minutes.\";\n ```\n\n ...would print:\n\n ```text\n Tea will be ready in 6 minutes.\n ```\n\n What token types would you define to implement a scanner for string\n interpolation? What sequence of tokens would you emit for the above string\n literal?\n\n What tokens would you emit for:\n\n ```text\n \"Nested ${\"interpolation?! Are you ${\"mad?!\"}\"}\"\n ```\n\n Consider looking at other language implementations that support\n interpolation to see how they handle it.\n\n2. Several languages use angle brackets for generics and also have a `>>` right\n shift operator. This led to a classic problem in early versions of C++:\n\n ```c++\n vector> nestedVectors;\n ```\n\n This would produce a compile error because the `>>` was lexed to a single\n right shift token, not two `>` tokens. Users were forced to avoid this by\n putting a space between the closing angle brackets.\n\n Later versions of C++ are smarter and can handle the above code. Java and C#\n never had the problem. How do those languages specify and implement this?\n\n3. Many languages, especially later in their evolution, define \"contextual\n keywords\". These are identifiers that act like reserved words in some\n contexts but can be normal user-defined identifiers in others.\n\n For example, `await` is a keyword inside an `async` method in C#, but\n in other methods, you can use `await` as your own identifier.\n\n Name a few contextual keywords from other languages, and the context where\n they are meaningful. What are the pros and cons of having contextual\n keywords? How would you implement them in your language's front end if you\n needed to?\n\n[interp]: https://en.wikipedia.org/wiki/String_interpolation\n\n
\n" }, { "path": "book/scanning.md", "content": "> Take big bites. Anything worth doing is worth overdoing.\n>\n> Robert A. Heinlein, Time Enough for Love\n\nThe first step in any compiler or interpreter is scanning. The scanner takes in raw source code as a series\nof characters and groups it into a series of chunks we call **tokens**. These\nare the meaningful \"words\" and \"punctuation\" that make up the language's\ngrammar.\n\n\n\nScanning is a good starting point for us too because the code isn't very hard --\npretty much a `switch` statement with delusions of grandeur. It will help us\nwarm up before we tackle some of the more interesting material later. By the end\nof this chapter, we'll have a full-featured, fast scanner that can take any\nstring of Lox source code and produce the tokens that we'll feed into the parser\nin the next chapter.\n\n## The Interpreter Framework\n\nSince this is our first real chapter, before we get to actually scanning some\ncode we need to sketch out the basic shape of our interpreter, jlox. Everything\nstarts with a class in Java.\n\n^code lox-class\n\n\n\nStick that in a text file, and go get your IDE or Makefile or whatever set up.\nI'll be right here when you're ready. Good? OK!\n\nLox is a scripting language, which means it executes directly from source. Our\ninterpreter supports two ways of running code. If you start jlox from the\ncommand line and give it a path to a file, it reads the file and executes it.\n\n^code run-file\n\nIf you want a more intimate conversation with your interpreter, you can also run\nit interactively. Fire up jlox without any arguments, and it drops you into a\nprompt where you can enter and execute code one line at a time.\n\n\n\n^code prompt\n\nThe `readLine()` function, as the name so helpfully implies, reads a line of\ninput from the user on the command line and returns the result. To kill an\ninteractive command-line app, you usually type Control-D. Doing so signals an\n\"end-of-file\" condition to the program. When that happens `readLine()` returns\n`null`, so we check for that to exit the loop.\n\nBoth the prompt and the file runner are thin wrappers around this core function:\n\n^code run\n\nIt's not super useful yet since we haven't written the interpreter, but baby\nsteps, you know? Right now, it prints out the tokens our forthcoming scanner\nwill emit so that we can see if we're making progress.\n\n### Error handling\n\nWhile we're setting things up, another key piece of infrastructure is *error\nhandling*. Textbooks sometimes gloss over this because it's more a practical\nmatter than a formal computer science-y problem. But if you care about making a\nlanguage that's actually *usable*, then handling errors gracefully is vital.\n\nThe tools our language provides for dealing with errors make up a large portion\nof its user interface. When the user's code is working, they aren't thinking\nabout our language at all -- their headspace is all about *their program*. It's\nusually only when things go wrong that they notice our implementation.\n\nWhen that happens, it's up to us to give the user all\nthe information they need to understand what went wrong and guide them gently\nback to where they are trying to go. Doing that well means thinking about error\nhandling all through the implementation of our interpreter, starting now.\n\n\n\n^code lox-error\n\nThis `error()` function and its `report()` helper tells the user some syntax\nerror occurred on a given line. That is really the bare minimum to be able to\nclaim you even *have* error reporting. Imagine if you accidentally left a\ndangling comma in some function call and the interpreter printed out:\n\n```text\nError: Unexpected \",\" somewhere in your code. Good luck finding it!\n```\n\nThat's not very helpful. We need to at least point them to the right line. Even\nbetter would be the beginning and end column so they know *where* in the line.\nEven better than *that* is to *show* the user the offending line, like:\n\n```text\nError: Unexpected \",\" in argument list.\n\n 15 | function(first, second,);\n ^-- Here.\n```\n\nI'd love to implement something like that in this book but the honest truth is\nthat it's a lot of grungy string manipulation code. Very useful for users, but\nnot super fun to read in a book and not very technically interesting. So we'll\nstick with just a line number. In your own interpreters, please do as I say and\nnot as I do.\n\nThe primary reason we're sticking this error reporting function in the main Lox\nclass is because of that `hadError` field. It's defined here:\n\n^code had-error (1 before)\n\nWe'll use this to ensure we don't try to execute code that has a known error.\nAlso, it lets us exit with a non-zero exit code like a good command line citizen\nshould.\n\n^code exit-code (1 before, 1 after)\n\nWe need to reset this flag in the interactive loop. If the user makes a mistake,\nit shouldn't kill their entire session.\n\n^code reset-had-error (1 before, 1 after)\n\nThe other reason I pulled the error reporting out here instead of stuffing it\ninto the scanner and other phases where the error might occur is to remind you\nthat it's good engineering practice to separate the code that *generates* the\nerrors from the code that *reports* them.\n\nVarious phases of the front end will detect errors, but it's not really their\njob to know how to present that to a user. In a full-featured language\nimplementation, you will likely have multiple ways errors get displayed: on\nstderr, in an IDE's error window, logged to a file, etc. You don't want that\ncode smeared all over your scanner and parser.\n\nIdeally, we would have an actual abstraction, some kind of \"ErrorReporter\" interface that gets passed to the scanner\nand parser so that we can swap out different reporting strategies. For our\nsimple interpreter here, I didn't do that, but I did at least move the code for\nerror reporting into a different class.\n\n\n\nWith some rudimentary error handling in place, our application shell is ready.\nOnce we have a Scanner class with a `scanTokens()` method, we can start running\nit. Before we get to that, let's get more precise about what tokens are.\n\n## Lexemes and Tokens\n\nHere's a line of Lox code:\n\n```lox\nvar language = \"lox\";\n```\n\nHere, `var` is the keyword for declaring a variable. That three-character\nsequence \"v-a-r\" means something. But if we yank three letters out of the\nmiddle of `language`, like \"g-u-a\", those don't mean anything on their own.\n\nThat's what lexical analysis is about. Our job is to scan through the list of\ncharacters and group them together into the smallest sequences that still\nrepresent something. Each of these blobs of characters is called a **lexeme**.\nIn that example line of code, the lexemes are:\n\n\"'var',\n\nThe lexemes are only the raw substrings of the source code. However, in the\nprocess of grouping character sequences into lexemes, we also stumble upon some\nother useful information. When we take the lexeme and bundle it together with\nthat other data, the result is a token. It includes useful stuff like:\n\n### Token type\n\nKeywords are part of the shape of the language's grammar, so the parser often\nhas code like, \"If the next token is `while` then do...\" That means the parser\nwants to know not just that it has a lexeme for some identifier, but that it has\na *reserved* word, and *which* keyword it is.\n\nThe parser could categorize tokens from the raw lexeme\nby comparing the strings, but that's slow and kind of ugly. Instead, at the\npoint that we recognize a lexeme, we also remember which *kind* of lexeme it\nrepresents. We have a different type for each keyword, operator, bit of\npunctuation, and literal type.\n\n\n\n^code token-type\n\n### Literal value\n\nThere are lexemes for literal values -- numbers and strings and the like. Since\nthe scanner has to walk each character in the literal to correctly identify it,\nit can also convert that textual representation of a value to the living runtime\nobject that will be used by the interpreter later.\n\n### Location information\n\nBack when I was preaching the gospel about error handling, we saw that we need\nto tell users *where* errors occurred. Tracking that starts here. In our simple\ninterpreter, we note only which line the token appears on, but more\nsophisticated implementations include the column and length too.\n\n\n\nWe take all of this data and wrap it in a class.\n\n^code token-class\n\nNow we have an object with enough structure to be useful for all of the later\nphases of the interpreter.\n\n## Regular Languages and Expressions\n\nNow that we know what we're trying to produce, let's, well, produce it. The core\nof the scanner is a loop. Starting at the first character of the source code,\nthe scanner figures out what lexeme the character belongs to, and consumes it\nand any following characters that are part of that lexeme. When it reaches the\nend of that lexeme, it emits a token.\n\nThen it loops back and does it again, starting from the very next character in\nthe source code. It keeps doing that, eating characters and occasionally, uh,\nexcreting tokens, until it reaches the end of the input.\n\n\n\n\"An\n\n\n\nThe part of the loop where we look at a handful of characters to figure out\nwhich kind of lexeme it \"matches\" may sound familiar. If you know regular\nexpressions, you might consider defining a regex for each kind of lexeme and\nusing those to match characters. For example, Lox has the same rules as C for\nidentifiers (variable names and the like). This regex matches one:\n\n```text\n[a-zA-Z_][a-zA-Z_0-9]*\n```\n\nIf you did think of regular expressions, your intuition is a deep one. The rules\nthat determine how a particular language groups characters into lexemes are\ncalled its **lexical grammar**. In Lox, as in most\nprogramming languages, the rules of that grammar are simple enough for the\nlanguage to be classified a **[regular language][]**. That's the same \"regular\"\nas in regular expressions.\n\n[regular language]: https://en.wikipedia.org/wiki/Regular_language\n\n\n\nYou very precisely *can* recognize all of the different lexemes for Lox using\nregexes if you want to, and there's a pile of interesting theory underlying why\nthat is and what it means. Tools like [Lex][] or\n[Flex][] are designed expressly to let you do this -- throw a handful of regexes\nat them, and they give you a complete scanner back.\n\n\n\n[lex]: http://dinosaur.compilertools.net/lex/\n[flex]: https://github.com/westes/flex\n\nSince our goal is to understand how a scanner does what it does, we won't be\ndelegating that task. We're about handcrafted goods.\n\n## The Scanner Class\n\nWithout further ado, let's make ourselves a scanner.\n\n^code scanner-class\n\n\n\nWe store the raw source code as a simple string, and we have a list ready to\nfill with tokens we're going to generate. The aforementioned loop that does that\nlooks like this:\n\n^code scan-tokens\n\nThe scanner works its way through the source code, adding tokens until it runs\nout of characters. Then it appends one final \"end of file\" token. That isn't\nstrictly needed, but it makes our parser a little cleaner.\n\nThis loop depends on a couple of fields to keep track of where the scanner is in\nthe source code.\n\n^code scan-state (1 before, 2 after)\n\nThe `start` and `current` fields are offsets that index into the string. The\n`start` field points to the first character in the lexeme being scanned, and\n`current` points at the character currently being considered. The `line` field\ntracks what source line `current` is on so we can produce tokens that know their\nlocation.\n\nThen we have one little helper function that tells us if we've consumed all the\ncharacters.\n\n^code is-at-end\n\n## Recognizing Lexemes\n\nIn each turn of the loop, we scan a single token. This is the real heart of the\nscanner. We'll start simple. Imagine if every lexeme were only a single character\nlong. All you would need to do is consume the next character and pick a token type for\nit. Several lexemes *are* only a single character in Lox, so let's start with\nthose.\n\n^code scan-token\n\n\n\nAgain, we need a couple of helper methods.\n\n^code advance-and-add-token\n\nThe `advance()` method consumes the next character in the source file and\nreturns it. Where `advance()` is for input, `addToken()` is for output. It grabs\nthe text of the current lexeme and creates a new token for it. We'll use the\nother overload to handle tokens with literal values soon.\n\n### Lexical errors\n\nBefore we get too far in, let's take a moment to think about errors at the\nlexical level. What happens if a user throws a source file containing some\ncharacters Lox doesn't use, like `@#^`, at our interpreter? Right now, those\ncharacters get silently discarded. They aren't used by the Lox language, but\nthat doesn't mean the interpreter can pretend they aren't there. Instead, we\nreport an error.\n\n^code char-error (1 before, 1 after)\n\nNote that the erroneous character is still *consumed* by the earlier call to\n`advance()`. That's important so that we don't get stuck in an infinite loop.\n\nNote also that we *keep scanning*. There may be\nother errors later in the program. It gives our users a better experience if we\ndetect as many of those as possible in one go. Otherwise, they see one tiny\nerror and fix it, only to have the next error appear, and so on. Syntax error\nWhac-A-Mole is no fun.\n\n(Don't worry. Since `hadError` gets set, we'll never try to *execute* any of the\ncode, even though we keep going and scan the rest of it.)\n\n\n\n### Operators\n\nWe have single-character lexemes working, but that doesn't cover all of Lox's\noperators. What about `!`? It's a single character, right? Sometimes, yes, but\nif the very next character is an equals sign, then we should instead create a\n`!=` lexeme. Note that the `!` and `=` are *not* two independent operators. You\ncan't write `! =` in Lox and have it behave like an inequality operator.\nThat's why we need to scan `!=` as a single lexeme. Likewise, `<`, `>`, and `=`\ncan all be followed by `=` to create the other equality and comparison\noperators.\n\nFor all of these, we need to look at the second character.\n\n^code two-char-tokens (1 before, 2 after)\n\nThose cases use this new method:\n\n^code match\n\nIt's like a conditional `advance()`. We only consume the current character if\nit's what we're looking for.\n\nUsing `match()`, we recognize these lexemes in two stages. When we reach, for\nexample, `!`, we jump to its switch case. That means we know the lexeme *starts*\nwith `!`. Then we look at the next character to determine if we're on a `!=` or\nmerely a `!`.\n\n## Longer Lexemes\n\nWe're still missing one operator: `/` for division. That character needs a\nlittle special handling because comments begin with a slash too.\n\n^code slash (1 before, 2 after)\n\nThis is similar to the other two-character operators, except that when we find a\nsecond `/`, we don't end the token yet. Instead, we keep consuming characters\nuntil we reach the end of the line.\n\nThis is our general strategy for handling longer lexemes. After we detect the\nbeginning of one, we shunt over to some lexeme-specific code that keeps eating\ncharacters until it sees the end.\n\nWe've got another helper:\n\n^code peek\n\nIt's sort of like `advance()`, but doesn't consume the character. This is called\n**lookahead**. Since it only looks at the current\nunconsumed character, we have *one character of lookahead*. The smaller this\nnumber is, generally, the faster the scanner runs. The rules of the lexical\ngrammar dictate how much lookahead we need. Fortunately, most languages in wide\nuse peek only one or two characters ahead.\n\n\n\nComments are lexemes, but they aren't meaningful, and the parser doesn't want\nto deal with them. So when we reach the end of the comment, we *don't* call\n`addToken()`. When we loop back around to start the next lexeme, `start` gets\nreset and the comment's lexeme disappears in a puff of smoke.\n\nWhile we're at it, now's a good time to skip over those other meaningless\ncharacters: newlines and whitespace.\n\n^code whitespace (1 before, 3 after)\n\nWhen encountering whitespace, we simply go back to the beginning of the scan\nloop. That starts a new lexeme *after* the whitespace character. For newlines,\nwe do the same thing, but we also increment the line counter. (This is why we\nused `peek()` to find the newline ending a comment instead of `match()`. We want\nthat newline to get us here so we can update `line`.)\n\nOur scanner is getting smarter. It can handle fairly free-form code like:\n\n```lox\n// this is a comment\n(( )){} // grouping stuff\n!*+-/=<> <= == // operators\n```\n\n### String literals\n\nNow that we're comfortable with longer lexemes, we're ready to tackle literals.\nWe'll do strings first, since they always begin with a specific character, `\"`.\n\n^code string-start (1 before, 2 after)\n\nThat calls:\n\n^code string\n\nLike with comments, we consume characters until we hit the `\"` that ends the\nstring. We also gracefully handle running out of input before the string is\nclosed and report an error for that.\n\nFor no particular reason, Lox supports multi-line strings. There are pros and\ncons to that, but prohibiting them was a little more complex than allowing them,\nso I left them in. That does mean we also need to update `line` when we hit a\nnewline inside a string.\n\nFinally, the last interesting bit is that when we create the token, we also\nproduce the actual string *value* that will be used later by the interpreter.\nHere, that conversion only requires a `substring()` to strip off the surrounding\nquotes. If Lox supported escape sequences like `\\n`, we'd unescape those here.\n\n### Number literals\n\nAll numbers in Lox are floating point at runtime, but both integer and decimal\nliterals are supported. A number literal is a series of digits optionally followed by a `.` and one or more trailing\ndigits.\n\n\n\n```lox\n1234\n12.34\n```\n\nWe don't allow a leading or trailing decimal point, so these are both invalid:\n\n```lox\n.1234\n1234.\n```\n\nWe could easily support the former, but I left it out to keep things simple. The\nlatter gets weird if we ever want to allow methods on numbers like `123.sqrt()`.\n\nTo recognize the beginning of a number lexeme, we look for any digit. It's kind\nof tedious to add cases for every decimal digit, so we'll stuff it in the\ndefault case instead.\n\n^code digit-start (1 before, 1 after)\n\nThis relies on this little utility:\n\n^code is-digit\n\n\n\nOnce we know we are in a number, we branch to a separate method to consume the\nrest of the literal, like we do with strings.\n\n^code number\n\nWe consume as many digits as we find for the integer part of the literal. Then\nwe look for a fractional part, which is a decimal point (`.`) followed by at\nleast one digit. If we do have a fractional part, again, we consume as many\ndigits as we can find.\n\nLooking past the decimal point requires a second character of lookahead since we\ndon't want to consume the `.` until we're sure there is a digit *after* it. So\nwe add:\n\n^code peek-next\n\n\n\n\nFinally, we convert the lexeme to its numeric value. Our interpreter uses Java's\n`Double` type to represent numbers, so we produce a value of that type. We're\nusing Java's own parsing method to convert the lexeme to a real Java double. We\ncould implement that ourselves, but, honestly, unless you're trying to cram for\nan upcoming programming interview, it's not worth your time.\n\nThe remaining literals are Booleans and `nil`, but we handle those as keywords,\nwhich gets us to...\n\n## Reserved Words and Identifiers\n\nOur scanner is almost done. The only remaining pieces of the lexical grammar to\nimplement are identifiers and their close cousins, the reserved words. You might\nthink we could match keywords like `or` in the same way we handle\nmultiple-character operators like `<=`.\n\n```java\ncase 'o':\n if (match('r')) {\n addToken(OR);\n }\n break;\n```\n\nConsider what would happen if a user named a variable `orchid`. The scanner\nwould see the first two letters, `or`, and immediately emit an `or` keyword\ntoken. This gets us to an important principle called **maximal munch**. When two lexical grammar rules can both\nmatch a chunk of code that the scanner is looking at, *whichever one matches the\nmost characters wins*.\n\nThat rule states that if we can match `orchid` as an identifier and `or` as a\nkeyword, then the former wins. This is also why we tacitly assumed, previously,\nthat `<=` should be scanned as a single `<=` token and not `<` followed by `=`.\n\n\n\nMaximal munch means we can't easily detect a reserved word until we've reached\nthe end of what might instead be an identifier. After all, a reserved word *is*\nan identifier, it's just one that has been claimed by the language for its own\nuse. That's where the term **reserved word** comes from.\n\nSo we begin by assuming any lexeme starting with a letter or underscore is an\nidentifier.\n\n^code identifier-start (3 before, 3 after)\n\nThe rest of the code lives over here:\n\n^code identifier\n\nWe define that in terms of these helpers:\n\n^code is-alpha\n\nThat gets identifiers working. To handle keywords, we see if the identifier's\nlexeme is one of the reserved words. If so, we use a token type specific to that\nkeyword. We define the set of reserved words in a map.\n\n^code keyword-map\n\nThen, after we scan an identifier, we check to see if it matches anything in the\nmap.\n\n^code keyword-type (2 before, 1 after)\n\nIf so, we use that keyword's token type. Otherwise, it's a regular user-defined\nidentifier.\n\nAnd with that, we now have a complete scanner for the entire Lox lexical\ngrammar. Fire up the REPL and type in some valid and invalid code. Does it\nproduce the tokens you expect? Try to come up with some interesting edge cases\nand see if it handles them as it should.\n\n
\n\n## Challenges\n\n1. The lexical grammars of Python and Haskell are not *regular*. What does that\n mean, and why aren't they?\n\n1. Aside from separating tokens -- distinguishing `print foo` from `printfoo`\n -- spaces aren't used for much in most languages. However, in a couple of\n dark corners, a space *does* affect how code is parsed in CoffeeScript,\n Ruby, and the C preprocessor. Where and what effect does it have in each of\n those languages?\n\n1. Our scanner here, like most, discards comments and whitespace since those\n aren't needed by the parser. Why might you want to write a scanner that does\n *not* discard those? What would it be useful for?\n\n1. Add support to Lox's scanner for C-style `/* ... */` block comments. Make\n sure to handle newlines in them. Consider allowing them to nest. Is adding\n support for nesting more work than you expected? Why?\n\n
\n\n
\n\n## Design Note: Implicit Semicolons\n\nProgrammers today are spoiled for choice in languages and have gotten picky\nabout syntax. They want their language to look clean and modern. One bit of\nsyntactic lichen that almost every new language scrapes off (and some ancient\nones like BASIC never had) is `;` as an explicit statement terminator.\n\nInstead, they treat a newline as a statement terminator where it makes sense to\ndo so. The \"where it makes sense\" part is the challenging bit. While *most*\nstatements are on their own line, sometimes you need to spread a single\nstatement across a couple of lines. Those intermingled newlines should not be\ntreated as terminators.\n\nMost of the obvious cases where the newline should be ignored are easy to\ndetect, but there are a handful of nasty ones:\n\n* A return value on the next line:\n\n ```js\n if (condition) return\n \"value\"\n ```\n\n Is \"value\" the value being returned, or do we have a `return` statement with\n no value followed by an expression statement containing a string literal?\n\n* A parenthesized expression on the next line:\n\n ```js\n func\n (parenthesized)\n ```\n\n Is this a call to `func(parenthesized)`, or two expression statements, one\n for `func` and one for a parenthesized expression?\n\n* A `-` on the next line:\n\n ```js\n first\n -second\n ```\n\n Is this `first - second` -- an infix subtraction -- or two expression\n statements, one for `first` and one to negate `second`?\n\nIn all of these, either treating the newline as a separator or not would both\nproduce valid code, but possibly not the code the user wants. Across languages,\nthere is an unsettling variety of rules used to decide which newlines are\nseparators. Here are a couple:\n\n* [Lua][] completely ignores newlines, but carefully controls its grammar such\n that no separator between statements is needed at all in most cases. This is\n perfectly legit:\n\n ```lua\n a = 1 b = 2\n ```\n\n Lua avoids the `return` problem by requiring a `return` statement to be the\n very last statement in a block. If there is a value after `return` before\n the keyword `end`, it *must* be for the `return`. For the other two cases,\n they allow an explicit `;` and expect users to use that. In practice, that\n almost never happens because there's no point in a parenthesized or unary\n negation expression statement.\n\n* [Go][] handles newlines in the scanner. If a newline appears following one\n of a handful of token types that are known to potentially end a statement,\n the newline is treated like a semicolon. Otherwise it is ignored. The Go\n team provides a canonical code formatter, [gofmt][], and the ecosystem is\n fervent about its use, which ensures that idiomatic styled code works well\n with this simple rule.\n\n* [Python][] treats all newlines as significant unless an explicit backslash\n is used at the end of a line to continue it to the next line. However,\n newlines anywhere inside a pair of brackets (`()`, `[]`, or `{}`) are\n ignored. Idiomatic style strongly prefers the latter.\n\n This rule works well for Python because it is a highly statement-oriented\n language. In particular, Python's grammar ensures a statement never appears\n inside an expression. C does the same, but many other languages which have a\n \"lambda\" or function literal syntax do not.\n\n An example in JavaScript:\n\n ```js\n console.log(function() {\n statement();\n });\n ```\n\n Here, the `console.log()` *expression* contains a function literal which\n in turn contains the *statement* `statement();`.\n\n Python would need a different set of rules for implicitly joining lines if\n you could get back *into* a statement where\n newlines should become meaningful while still nested inside brackets.\n\n\n\n* JavaScript's \"[automatic semicolon insertion][asi]\" rule is the real odd\n one. Where other languages assume most newlines *are* meaningful and only a\n few should be ignored in multi-line statements, JS assumes the opposite. It\n treats all of your newlines as meaningless whitespace *unless* it encounters\n a parse error. If it does, it goes back and tries turning the previous\n newline into a semicolon to get something grammatically valid.\n\n This design note would turn into a design diatribe if I went into complete\n detail about how that even *works*, much less all the various ways that\n JavaScript's \"solution\" is a bad idea. It's a mess. JavaScript is the only\n language I know where many style guides demand explicit semicolons after\n every statement even though the language theoretically lets you elide them.\n\nIf you're designing a new language, you almost surely *should* avoid an explicit\nstatement terminator. Programmers are creatures of fashion like other humans, and\nsemicolons are as passé as ALL CAPS KEYWORDS. Just make sure you pick a set of\nrules that make sense for your language's particular grammar and idioms. And\ndon't do what JavaScript did.\n\n
\n\n[lua]: https://www.lua.org/pil/1.1.html\n[go]: https://golang.org/ref/spec#Semicolons\n[gofmt]: https://golang.org/cmd/gofmt/\n[python]: https://docs.python.org/3.5/reference/lexical_analysis.html#implicit-line-joining\n[asi]: https://www.ecma-international.org/ecma-262/5.1/#sec-7.9\n" }, { "path": "book/statements-and-state.md", "content": "> All my life, my heart has yearned for a thing I cannot name.\n> André Breton, Mad Love\n\nThe interpreter we have so far feels less like programming a real language and\nmore like punching buttons on a calculator. \"Programming\" to me means building\nup a system out of smaller pieces. We can't do that yet because we have no way\nto bind a name to some data or function. We can't compose software without a way\nto refer to the pieces.\n\nTo support bindings, our interpreter needs internal state. When you define a\nvariable at the beginning of the program and use it at the end, the interpreter\nhas to hold on to the value of that variable in the meantime. So in this\nchapter, we will give our interpreter a brain that can not just process, but\n*remember*.\n\n\"A\n\nState and statements go hand in hand. Since statements,\nby definition, don't evaluate to a value, they need to do something else to be\nuseful. That something is called a **side effect**. It could mean producing\nuser-visible output or modifying some state in the interpreter that can be\ndetected later. The latter makes them a great fit for defining variables or\nother named entities.\n\n\n\nIn this chapter, we'll do all of that. We'll define statements that produce\noutput (`print`) and create state (`var`). We'll add expressions to access and\nassign to variables. Finally, we'll add blocks and local scope. That's a lot to\nstuff into one chapter, but we'll chew through it all one bite at a time.\n\n## Statements\n\nWe start by extending Lox's grammar with statements. They aren't very different\nfrom expressions. We start with the two simplest kinds:\n\n1. An **expression statement** lets you place an expression where a statement\n is expected. They exist to evaluate expressions that have side effects. You\n may not notice them, but you use them all the time in C, Java, and other languages. Any time you see a\n function or method call followed by a `;`, you're looking at an expression\n statement.\n\n \n\n2. A **`print` statement** evaluates an expression and displays the result to\n the user. I admit it's weird to bake printing right into the language\n instead of making it a library function. Doing so is a concession to the\n fact that we're building this interpreter one chapter at a time and want to\n be able to play with it before it's all done. To make print a library\n function, we'd have to wait until we had all of the machinery for defining\n and calling functions before we could witness any\n side effects.\n\n \n\nNew syntax means new grammar rules. In this chapter, we finally gain the ability\nto parse an entire Lox script. Since Lox is an imperative, dynamically typed\nlanguage, the \"top level\" of a script is simply a list of statements. The new\nrules are:\n\n```ebnf\nprogram → statement* EOF ;\n\nstatement → exprStmt\n | printStmt ;\n\nexprStmt → expression \";\" ;\nprintStmt → \"print\" expression \";\" ;\n```\n\nThe first rule is now `program`, which is the starting point for the grammar and\nrepresents a complete Lox script or REPL entry. A program is a list of\nstatements followed by the special \"end of file\" token. The mandatory end token\nensures the parser consumes the entire input and doesn't silently ignore\nerroneous unconsumed tokens at the end of a script.\n\nRight now, `statement` only has two cases for the two kinds of statements we've\ndescribed. We'll fill in more later in this chapter and in the following ones.\nThe next step is turning this grammar into something we can store in memory --\nsyntax trees.\n\n### Statement syntax trees\n\nThere is no place in the grammar where both an expression and a statement are\nallowed. The operands of, say, `+` are always expressions, never statements. The\nbody of a `while` loop is always a statement.\n\nSince the two syntaxes are disjoint, we don't need a single base class that they\nall inherit from. Splitting expressions and statements into separate class\nhierarchies enables the Java compiler to help us find dumb mistakes like passing\na statement to a Java method that expects an expression.\n\nThat means a new base class for statements. As our elders did before us, we will\nuse the cryptic name \"Stmt\". With great foresight,\nI have designed our little AST metaprogramming script in anticipation of this.\nThat's why we passed in \"Expr\" as a parameter to `defineAst()`. Now we add\nanother call to define Stmt and its subclasses.\n\n\n\n^code stmt-ast (2 before, 1 after)\n\n\n\nRun the AST generator script and behold the resulting \"Stmt.java\" file with the\nsyntax tree classes we need for expression and `print` statements. Don't forget\nto add the file to your IDE project or makefile or whatever.\n\n### Parsing statements\n\nThe parser's `parse()` method that parses and returns a single expression was a\ntemporary hack to get the last chapter up and running. Now that our grammar has\nthe correct starting rule, `program`, we can turn `parse()` into the real deal.\n\n^code parse\n\n\n\nThis parses a series of statements, as many as it can find until it hits the end\nof the input. This is a pretty direct translation of the `program` rule into\nrecursive descent style. We must also chant a minor prayer to the Java verbosity\ngods since we are using ArrayList now.\n\n^code parser-imports (2 before, 1 after)\n\nA program is a list of statements, and we parse one of those statements using\nthis method:\n\n^code parse-statement\n\nA little bare bones, but we'll fill it in with more statement types later. We\ndetermine which specific statement rule is matched by looking at the current\ntoken. A `print` token means it's obviously a `print` statement.\n\nIf the next token doesn't look like any known kind of statement, we assume it\nmust be an expression statement. That's the typical final fallthrough case when\nparsing a statement, since it's hard to proactively recognize an expression from\nits first token.\n\nEach statement kind gets its own method. First `print`:\n\n^code parse-print-statement\n\nSince we already matched and consumed the `print` token itself, we don't need to\ndo that here. We parse the subsequent expression, consume the terminating\nsemicolon, and emit the syntax tree.\n\nIf we didn't match a `print` statement, we must have one of these:\n\n^code parse-expression-statement\n\nSimilar to the previous method, we parse an expression followed by a semicolon.\nWe wrap that Expr in a Stmt of the right type and return it.\n\n### Executing statements\n\nWe're running through the previous couple of chapters in microcosm, working our\nway through the front end. Our parser can now produce statement syntax trees, so\nthe next and final step is to interpret them. As in expressions, we use the\nVisitor pattern, but we have a new visitor interface, Stmt.Visitor, to\nimplement since statements have their own base class.\n\nWe add that to the list of interfaces Interpreter implements.\n\n^code interpreter (1 after)\n\n\n\nUnlike expressions, statements produce no values, so the return type of the\nvisit methods is Void, not Object. We have two statement types, and we need a\nvisit method for each. The easiest is expression statements.\n\n^code visit-expression-stmt\n\nWe evaluate the inner expression using our existing `evaluate()` method and\ndiscard the value. Then we return `null`. Java\nrequires that to satisfy the special capitalized Void return type. Weird, but\nwhat can you do?\n\n\n\nThe `print` statement's visit method isn't much different.\n\n^code visit-print\n\nBefore discarding the expression's value, we convert it to a string using the\n`stringify()` method we introduced in the last chapter and then dump it to\nstdout.\n\nOur interpreter is able to visit statements now, but we have some work to do to\nfeed them to it. First, modify the old `interpret()` method in the Interpreter\nclass to accept a list of statements -- in other words, a program.\n\n^code interpret\n\nThis replaces the old code which took a single expression. The new code relies\non this tiny helper method:\n\n^code execute\n\nThat's the statement analogue to the `evaluate()` method we have for\nexpressions. Since we're working with lists now, we need to let Java know.\n\n^code import-list (2 before, 2 after)\n\nThe main Lox class is still trying to parse a single expression and pass it to\nthe interpreter. We fix the parsing line like so:\n\n^code parse-statements (1 before, 2 after)\n\nAnd then replace the call to the interpreter with this:\n\n^code interpret-statements (2 before, 1 after)\n\nBasically just plumbing the new syntax through. OK, fire up the interpreter and\ngive it a try. At this point, it's worth sketching out a little Lox program in a\ntext file to run as a script. Something like:\n\n```lox\nprint \"one\";\nprint true;\nprint 2 + 1;\n```\n\nIt almost looks like a real program! Note that the REPL, too, now requires you\nto enter a full statement instead of a simple expression. Don't forget your\nsemicolons.\n\n## Global Variables\n\nNow that we have statements, we can start working on state. Before we get into\nall of the complexity of lexical scoping, we'll start off with the easiest kind\nof variables -- globals. We need two new constructs.\n\n1. A **variable declaration** statement brings a new variable into the world.\n\n ```lox\n var beverage = \"espresso\";\n ```\n\n This creates a new binding that associates a name (here \"beverage\") with a\n value (here, the string `\"espresso\"`).\n\n2. Once that's done, a **variable expression** accesses that binding. When the\n identifier \"beverage\" is used as an expression, it looks up the value bound\n to that name and returns it.\n\n ```lox\n print beverage; // \"espresso\".\n ```\n\nLater, we'll add assignment and block scope, but that's enough to get moving.\n\n\n\n### Variable syntax\n\nAs before, we'll work through the implementation from front to back, starting\nwith the syntax. Variable declarations are statements, but they are different\nfrom other statements, and we're going to split the statement grammar in two to\nhandle them. That's because the grammar restricts where some kinds of statements\nare allowed.\n\nThe clauses in control flow statements -- think the then and else branches of\nan `if` statement or the body of a `while` -- are each a single statement. But\nthat statement is not allowed to be one that declares a name. This is OK:\n\n```lox\nif (monday) print \"Ugh, already?\";\n```\n\nBut this is not:\n\n```lox\nif (monday) var beverage = \"espresso\";\n```\n\nWe *could* allow the latter, but it's confusing. What is the scope of that\n`beverage` variable? Does it persist after the `if` statement? If so, what is\nits value on days other than Monday? Does the variable exist at all on those\ndays?\n\nCode like this is weird, so C, Java, and friends all disallow it. It's as if\nthere are two levels of \"precedence\" for statements.\nSome places where a statement is allowed -- like inside a block or at the top\nlevel -- allow any kind of statement, including declarations. Others allow only\nthe \"higher\" precedence statements that don't declare names.\n\n\n\nTo accommodate the distinction, we add another rule for kinds of statements that\ndeclare names.\n\n```ebnf\nprogram → declaration* EOF ;\n\ndeclaration → varDecl\n | statement ;\n\nstatement → exprStmt\n | printStmt ;\n```\n\nDeclaration statements go under the new `declaration` rule. Right now, it's only\nvariables, but later it will include functions and classes. Any place where a\ndeclaration is allowed also allows non-declaring statements, so the\n`declaration` rule falls through to `statement`. Obviously, you can declare\nstuff at the top level of a script, so `program` routes to the new rule.\n\nThe rule for declaring a variable looks like:\n\n```ebnf\nvarDecl → \"var\" IDENTIFIER ( \"=\" expression )? \";\" ;\n```\n\nLike most statements, it starts with a leading keyword. In this case, `var`.\nThen an identifier token for the name of the variable being declared, followed\nby an optional initializer expression. Finally, we put a bow on it with the\nsemicolon.\n\nTo access a variable, we define a new kind of primary expression.\n\n```ebnf\nprimary → \"true\" | \"false\" | \"nil\"\n | NUMBER | STRING\n | \"(\" expression \")\"\n | IDENTIFIER ;\n```\n\nThat `IDENTIFIER` clause matches a single identifier token, which is understood\nto be the name of the variable being accessed.\n\nThese new grammar rules get their corresponding syntax trees. Over in the AST\ngenerator, we add a new statement node for a\nvariable declaration.\n\n^code var-stmt-ast (1 before, 1 after)\n\n\n\nIt stores the name token so we know what it's declaring, along with the\ninitializer expression. (If there isn't an initializer, that field is `null`.)\n\nThen we add an expression node for accessing a variable.\n\n^code var-expr (1 before, 1 after)\n\nIt's simply a wrapper around the token for the\nvariable name. That's it. As always, don't forget to run the AST generator\nscript so that you get updated \"Expr.java\" and \"Stmt.java\" files.\n\n\n\n### Parsing variables\n\nBefore we parse variable statements, we need to shift around some code to make\nroom for the new `declaration` rule in the grammar. The top level of a program\nis now a list of declarations, so the entrypoint method to the parser changes.\n\n^code parse-declaration (3 before, 4 after)\n\nThat calls this new method:\n\n^code declaration\n\nHey, do you remember way back in that [earlier chapter][parsing] when we put the\ninfrastructure in place to do error recovery? We are finally ready to hook that\nup.\n\n[parsing]: parsing-expressions.html\n[error recovery]: parsing-expressions.html#panic-mode-error-recovery\n\nThis `declaration()` method is the method we call repeatedly when parsing a\nseries of statements in a block or a script, so it's the right place to\nsynchronize when the parser goes into panic mode. The whole body of this method\nis wrapped in a try block to catch the exception thrown when the parser begins\nerror recovery. This gets it back to trying to parse the beginning of the next\nstatement or declaration.\n\nThe real parsing happens inside the try block. First, it looks to see if we're\nat a variable declaration by looking for the leading `var` keyword. If not, it\nfalls through to the existing `statement()` method that parses `print` and\nexpression statements.\n\nRemember how `statement()` tries to parse an expression statement if no other\nstatement matches? And `expression()` reports a syntax error if it can't parse\nan expression at the current token? That chain of calls ensures we report an\nerror if a valid declaration or statement isn't parsed.\n\nWhen the parser matches a `var` token, it branches to:\n\n^code parse-var-declaration\n\nAs always, the recursive descent code follows the grammar rule. The parser has\nalready matched the `var` token, so next it requires and consumes an identifier\ntoken for the variable name.\n\nThen, if it sees an `=` token, it knows there is an initializer expression and\nparses it. Otherwise, it leaves the initializer `null`. Finally, it consumes the\nrequired semicolon at the end of the statement. All this gets wrapped in a\nStmt.Var syntax tree node and we're groovy.\n\nParsing a variable expression is even easier. In `primary()`, we look for an\nidentifier token.\n\n^code parse-identifier (2 before, 2 after)\n\nThat gives us a working front end for declaring and using variables. All that's\nleft is to feed it into the interpreter. Before we get to that, we need to talk\nabout where variables live in memory.\n\n## Environments\n\nThe bindings that associate variables to values need to be stored somewhere.\nEver since the Lisp folks invented parentheses, this data structure has been\ncalled an **environment**.\n\n\"An\n\n\n\nYou can think of it like a map where the keys are\nvariable names and the values are the variable's, uh, values. In fact, that's\nhow we'll implement it in Java. We could stuff that map and the code to manage\nit right into Interpreter, but since it forms a nicely delineated concept, we'll\npull it out into its own class.\n\nStart a new file and add:\n\n\n\n^code environment-class\n\nThere's a Java Map in there to store the bindings. It uses bare strings for the\nkeys, not tokens. A token represents a unit of code at a specific place in the\nsource text, but when it comes to looking up variables, all identifier tokens\nwith the same name should refer to the same variable (ignoring scope for now).\nUsing the raw string ensures all of those tokens refer to the same map key.\n\nThere are two operations we need to support. First, a variable definition binds\na new name to a value.\n\n^code environment-define\n\nNot exactly brain surgery, but we have made one interesting semantic choice.\nWhen we add the key to the map, we don't check to see if it's already present.\nThat means that this program works:\n\n```lox\nvar a = \"before\";\nprint a; // \"before\".\nvar a = \"after\";\nprint a; // \"after\".\n```\n\nA variable statement doesn't just define a *new* variable, it can also be used\nto *re*define an existing variable. We could choose\nto make this an error instead. The user may not intend to redefine an existing\nvariable. (If they did mean to, they probably would have used assignment, not\n`var`.) Making redefinition an error would help them find that bug.\n\nHowever, doing so interacts poorly with the REPL. In the middle of a REPL\nsession, it's nice to not have to mentally track which variables you've already\ndefined. We could allow redefinition in the REPL but not in scripts, but then\nusers would have to learn two sets of rules, and code copied and pasted from one\nform to the other might not work.\n\n\n\nSo, to keep the two modes consistent, we'll allow it -- at least for global\nvariables. Once a variable exists, we need a way to look it up.\n\n^code environment-get (2 before, 1 after)\n\nThis is a little more semantically interesting. If the variable is found, it\nsimply returns the value bound to it. But what if it's not? Again, we have a\nchoice:\n\n* Make it a syntax error.\n\n* Make it a runtime error.\n\n* Allow it and return some default value like `nil`.\n\nLox is pretty lax, but the last option is a little *too* permissive to me.\nMaking it a syntax error -- a compile-time error -- seems like a smart choice.\nUsing an undefined variable is a bug, and the sooner you detect the mistake, the\nbetter.\n\nThe problem is that *using* a variable isn't the same as *referring* to it. You\ncan refer to a variable in a chunk of code without immediately evaluating it if\nthat chunk of code is wrapped inside a function. If we make it a static error to\n*mention* a variable before it's been declared, it becomes much harder to define\nrecursive functions.\n\nWe could accommodate single recursion -- a function that calls itself -- by\ndeclaring the function's own name before we examine its body. But that doesn't\nhelp with mutually recursive procedures that call each other. Consider:\n\n\n\n```lox\nfun isOdd(n) {\n if (n == 0) return false;\n return isEven(n - 1);\n}\n\nfun isEven(n) {\n if (n == 0) return true;\n return isOdd(n - 1);\n}\n```\n\n\n\nThe `isEven()` function isn't defined by the time we\nare looking at the body of `isOdd()` where it's called. If we swap the order of\nthe two functions, then `isOdd()` isn't defined when we're looking at\n`isEven()`'s body.\n\n\n\nSince making it a *static* error makes recursive declarations too difficult,\nwe'll defer the error to runtime. It's OK to refer to a variable before it's\ndefined as long as you don't *evaluate* the reference. That lets the program\nfor even and odd numbers work, but you'd get a runtime error in:\n\n```lox\nprint a;\nvar a = \"too late!\";\n```\n\nAs with type errors in the expression evaluation code, we report a runtime error\nby throwing an exception. The exception contains the variable's token so we can\ntell the user where in their code they messed up.\n\n### Interpreting global variables\n\nThe Interpreter class gets an instance of the new Environment class.\n\n^code environment-field (2 before, 1 after)\n\nWe store it as a field directly in Interpreter so that the variables stay in\nmemory as long as the interpreter is still running.\n\nWe have two new syntax trees, so that's two new visit methods. The first is for\ndeclaration statements.\n\n^code visit-var\n\nIf the variable has an initializer, we evaluate it. If not, we have another\nchoice to make. We could have made this a syntax error in the parser by\n*requiring* an initializer. Most languages don't, though, so it feels a little\nharsh to do so in Lox.\n\nWe could make it a runtime error. We'd let you define an uninitialized variable,\nbut if you accessed it before assigning to it, a runtime error would occur. It's\nnot a bad idea, but most dynamically typed languages don't do that. Instead,\nwe'll keep it simple and say that Lox sets a variable to `nil` if it isn't\nexplicitly initialized.\n\n```lox\nvar a;\nprint a; // \"nil\".\n```\n\nThus, if there isn't an initializer, we set the value to `null`, which is the\nJava representation of Lox's `nil` value. Then we tell the environment to bind\nthe variable to that value.\n\nNext, we evaluate a variable expression.\n\n^code visit-variable\n\nThis simply forwards to the environment which does the heavy lifting to make\nsure the variable is defined. With that, we've got rudimentary variables\nworking. Try this out:\n\n```lox\nvar a = 1;\nvar b = 2;\nprint a + b;\n```\n\nWe can't reuse *code* yet, but we can start to build up programs that reuse\n*data*.\n\n## Assignment\n\nIt's possible to create a language that has variables but does not let you\nreassign -- or **mutate** -- them. Haskell is one example. SML supports only\nmutable references and arrays -- variables cannot be reassigned. Rust steers you\naway from mutation by requiring a `mut` modifier to enable assignment.\n\nMutating a variable is a side effect and, as the name suggests, some language\nfolks think side effects are dirty or inelegant. Code\nshould be pure math that produces values -- crystalline, unchanging ones -- like\nan act of divine creation. Not some grubby automaton that beats blobs of data\ninto shape, one imperative grunt at a time.\n\n\n\nLox is not so austere. Lox is an imperative language, and mutation comes with\nthe territory. Adding support for assignment doesn't require much work. Global\nvariables already support redefinition, so most of the machinery is there now.\nMainly, we're missing an explicit assignment notation.\n\n### Assignment syntax\n\nThat little `=` syntax is more complex than it might seem. Like most C-derived\nlanguages, assignment is an expression and not a\nstatement. As in C, it is the lowest precedence expression form. That means the\nrule slots between `expression` and `equality` (the next lowest precedence\nexpression).\n\n\n\n```ebnf\nexpression → assignment ;\nassignment → IDENTIFIER \"=\" assignment\n | equality ;\n```\n\nThis says an `assignment` is either an identifier followed by an `=` and an\nexpression for the value, or an `equality` (and thus any other) expression.\nLater, `assignment` will get more complex when we add property setters on\nobjects, like:\n\n```lox\ninstance.field = \"value\";\n```\n\nThe easy part is adding the new syntax tree node.\n\n^code assign-expr (1 before, 1 after)\n\n\n\nIt has a token for the variable being assigned to, and an expression for the new\nvalue. After you run the AstGenerator to get the new Expr.Assign class, swap out\nthe body of the parser's existing `expression()` method to match the updated\nrule.\n\n^code expression (1 before, 1 after)\n\nHere is where it gets tricky. A single token lookahead recursive descent parser\ncan't see far enough to tell that it's parsing an assignment until *after* it\nhas gone through the left-hand side and stumbled onto the `=`. You might wonder\nwhy it even needs to. After all, we don't know we're parsing a `+` expression\nuntil after we've finished parsing the left operand.\n\nThe difference is that the left-hand side of an assignment isn't an expression\nthat evaluates to a value. It's a sort of pseudo-expression that evaluates to a\n\"thing\" you can assign to. Consider:\n\n```lox\nvar a = \"before\";\na = \"value\";\n```\n\nOn the second line, we don't *evaluate* `a` (which would return the string\n\"before\"). We figure out what variable `a` refers to so we know where to store\nthe right-hand side expression's value. The [classic terms][l-value] for these\ntwo constructs are **l-value** and **r-value**. All\nof the expressions that we've seen so far that produce values are r-values. An\nl-value \"evaluates\" to a storage location that you can assign into.\n\n[l-value]: https://en.wikipedia.org/wiki/Value_(computer_science)#lrvalue\n\n\n\nWe want the syntax tree to reflect that an l-value isn't evaluated like a normal\nexpression. That's why the Expr.Assign node has a *Token* for the left-hand\nside, not an Expr. The problem is that the parser doesn't know it's parsing an\nl-value until it hits the `=`. In a complex l-value, that may occur many tokens later.\n\n```lox\nmakeList().head.next = node;\n```\n\n\n\nWe have only a single token of lookahead, so what do we do? We use a little\ntrick, and it looks like this:\n\n^code parse-assignment\n\nMost of the code for parsing an assignment expression looks similar to that of\nthe other binary operators like `+`. We parse the left-hand side, which can be\nany expression of higher precedence. If we find an `=`, we parse the right-hand\nside and then wrap it all up in an assignment expression tree node.\n\n\n\nOne slight difference from binary operators is that we don't loop to build up a\nsequence of the same operator. Since assignment is right-associative, we instead\nrecursively call `assignment()` to parse the right-hand side.\n\nThe trick is that right before we create the assignment expression node, we look\nat the left-hand side expression and figure out what kind of assignment target\nit is. We convert the r-value expression node into an l-value representation.\n\nThis conversion works because it turns out that every valid assignment target\nhappens to also be valid syntax as a normal\nexpression. Consider a complex field assignment like:\n\n\n\n```lox\nnewPoint(x + 2, 0).y = 3;\n```\n\nThe left-hand side of that assignment could also work as a valid expression.\n\n```lox\nnewPoint(x + 2, 0).y;\n```\n\nThe first example sets the field, the second gets it.\n\nThis means we can parse the left-hand side *as if it were* an expression and\nthen after the fact produce a syntax tree that turns it into an assignment\ntarget. If the left-hand side expression isn't a valid\nassignment target, we fail with a syntax error. That ensures we report an error\non code like this:\n\n```lox\na + b = c;\n```\n\n\n\nRight now, the only valid target is a simple variable expression, but we'll add\nfields later. The end result of this trick is an assignment expression tree node\nthat knows what it is assigning to and has an expression subtree for the value\nbeing assigned. All with only a single token of lookahead and no backtracking.\n\n### Assignment semantics\n\nWe have a new syntax tree node, so our interpreter gets a new visit method.\n\n^code visit-assign\n\nFor obvious reasons, it's similar to variable declaration. It evaluates the\nright-hand side to get the value, then stores it in the named variable. Instead\nof using `define()` on Environment, it calls this new method:\n\n^code environment-assign\n\nThe key difference between assignment and definition is that assignment is not\nallowed to create a *new* variable. In terms of our\nimplementation, that means it's a runtime error if the key doesn't already exist\nin the environment's variable map.\n\n\n\nThe last thing the `visit()` method does is return the assigned value. That's\nbecause assignment is an expression that can be nested inside other expressions,\nlike so:\n\n```lox\nvar a = 1;\nprint a = 2; // \"2\".\n```\n\nOur interpreter can now create, read, and modify variables. It's about as\nsophisticated as early BASICs. Global variables are\nsimple, but writing a large program when any two chunks of code can accidentally\nstep on each other's state is no fun. We want *local* variables, which means\nit's time for *scope*.\n\n\n\n## Scope\n\nA **scope** defines a region where a name maps to a certain entity. Multiple\nscopes enable the same name to refer to different things in different contexts.\nIn my house, \"Bob\" usually refers to me. But maybe in your town you know a\ndifferent Bob. Same name, but different dudes based on where you say it.\n\n**Lexical scope** (or the less commonly heard\n**static scope**) is a specific style of scoping where the text of the program\nitself shows where a scope begins and ends. In Lox, as in most modern languages,\nvariables are lexically scoped. When you see an expression that uses some\nvariable, you can figure out which variable declaration it refers to just by\nstatically reading the code.\n\n\n\nFor example:\n\n```lox\n{\n var a = \"first\";\n print a; // \"first\".\n}\n\n{\n var a = \"second\";\n print a; // \"second\".\n}\n```\n\nHere, we have two blocks with a variable `a` declared in each of them. You and\nI can tell just from looking at the code that the use of `a` in the first\n`print` statement refers to the first `a`, and the second one refers to the\nsecond.\n\n\"An\n\nThis is in contrast to **dynamic scope** where you don't know what a name refers\nto until you execute the code. Lox doesn't have dynamically scoped *variables*,\nbut methods and fields on objects are dynamically scoped.\n\n```lox\nclass Saxophone {\n play() {\n print \"Careless Whisper\";\n }\n}\n\nclass GolfClub {\n play() {\n print \"Fore!\";\n }\n}\n\nfun playIt(thing) {\n thing.play();\n}\n```\n\nWhen `playIt()` calls `thing.play()`, we don't know if we're about to hear\n\"Careless Whisper\" or \"Fore!\" It depends on whether you pass a Saxophone or a\nGolfClub to the function, and we don't know that until runtime.\n\nScope and environments are close cousins. The former is the theoretical concept,\nand the latter is the machinery that implements it. As our interpreter works its\nway through code, syntax tree nodes that affect scope will change the\nenvironment. In a C-ish syntax like Lox's, scope is controlled by curly-braced\nblocks. (That's why we call it **block scope**.)\n\n```lox\n{\n var a = \"in block\";\n}\nprint a; // Error! No more \"a\".\n```\n\nThe beginning of a block introduces a new local scope, and that scope ends when\nexecution passes the closing `}`. Any variables declared inside the block\ndisappear.\n\n### Nesting and shadowing\n\nA first cut at implementing block scope might work like this:\n\n1. As we visit each statement inside the block, keep track of any variables\n declared.\n\n2. After the last statement is executed, tell the environment to delete all of\n those variables.\n\nThat would work for the previous example. But remember, one motivation for\nlocal scope is encapsulation -- a block of code in one corner of the program\nshouldn't interfere with some other block. Check this out:\n\n```lox\n// How loud?\nvar volume = 11;\n\n// Silence.\nvolume = 0;\n\n// Calculate size of 3x4x5 cuboid.\n{\n var volume = 3 * 4 * 5;\n print volume;\n}\n```\n\nLook at the block where we calculate the volume of the cuboid using a local\ndeclaration of `volume`. After the block exits, the interpreter will delete the\n*global* `volume` variable. That ain't right. When we exit the block, we should\nremove any variables declared inside the block, but if there is a variable with\nthe same name declared outside of the block, *that's a different variable*. It\nshouldn't get touched.\n\nWhen a local variable has the same name as a variable in an enclosing scope, it\n**shadows** the outer one. Code inside the block can't see it any more -- it is\nhidden in the \"shadow\" cast by the inner one -- but it's still there.\n\nWhen we enter a new block scope, we need to preserve variables defined in outer\nscopes so they are still around when we exit the inner block. We do that by\ndefining a fresh environment for each block containing only the variables\ndefined in that scope. When we exit the block, we discard its environment and\nrestore the previous one.\n\nWe also need to handle enclosing variables that are *not* shadowed.\n\n```lox\nvar global = \"outside\";\n{\n var local = \"inside\";\n print global + local;\n}\n```\n\nHere, `global` lives in the outer global environment and `local` is defined\ninside the block's environment. In that `print` statement, both of those\nvariables are in scope. In order to find them, the interpreter must search not\nonly the current innermost environment, but also any enclosing ones.\n\nWe implement this by chaining the environments\ntogether. Each environment has a reference to the environment of the immediately\nenclosing scope. When we look up a variable, we walk that chain from innermost\nout until we find the variable. Starting at the inner scope is how we make local\nvariables shadow outer ones.\n\n\"Environments\n\n\n\nBefore we add block syntax to the grammar, we'll beef up our Environment class\nwith support for this nesting. First, we give each environment a reference to\nits enclosing one.\n\n^code enclosing-field (1 before, 1 after)\n\nThis field needs to be initialized, so we add a couple of constructors.\n\n^code environment-constructors\n\nThe no-argument constructor is for the global scope's environment, which ends\nthe chain. The other constructor creates a new local scope nested inside the\ngiven outer one.\n\nWe don't have to touch the `define()` method -- a new variable is always\ndeclared in the current innermost scope. But variable lookup and assignment work\nwith existing variables and they need to walk the chain to find them. First,\nlookup:\n\n^code environment-get-enclosing (2 before, 3 after)\n\nIf the variable isn't found in this environment, we simply try the enclosing\none. That in turn does the same thing recursively,\nso this will ultimately walk the entire chain. If we reach an environment with\nno enclosing one and still don't find the variable, then we give up and report\nan error as before.\n\nAssignment works the same way.\n\n\n\n^code environment-assign-enclosing (4 before, 1 after)\n\nAgain, if the variable isn't in this environment, it checks the outer one,\nrecursively.\n\n### Block syntax and semantics\n\nNow that Environments nest, we're ready to add blocks to the language. Behold\nthe grammar:\n\n```ebnf\nstatement → exprStmt\n | printStmt\n | block ;\n\nblock → \"{\" declaration* \"}\" ;\n```\n\nA block is a (possibly empty) series of statements or declarations surrounded by\ncurly braces. A block is itself a statement and can appear anywhere a statement\nis allowed. The syntax tree node looks like this:\n\n^code block-ast (1 before, 1 after)\n\n\n\nIt contains the list of statements that are inside\nthe block. Parsing is straightforward. Like other statements, we detect the\nbeginning of a block by its leading token -- in this case the `{`. In the\n`statement()` method, we add:\n\n\n\n^code parse-block (1 before, 2 after)\n\nAll the real work happens here:\n\n^code block\n\nWe create an empty list and then parse statements and\nadd them to the list until we reach the end of the block, marked by the closing\n`}`. Note that the loop also has an explicit check for `isAtEnd()`. We have to\nbe careful to avoid infinite loops, even when parsing invalid code. If the user\nforgets a closing `}`, the parser needs to not get stuck.\n\n\n\nThat's it for syntax. For semantics, we add another visit method to Interpreter.\n\n^code visit-block\n\nTo execute a block, we create a new environment for the block's scope and pass\nit off to this other method:\n\n^code execute-block\n\nThis new method executes a list of statements in the context of a given environment. Up until now, the `environment` field in\nInterpreter always pointed to the same environment -- the global one. Now, that\nfield represents the *current* environment. That's the environment that\ncorresponds to the innermost scope containing the code to be executed.\n\nTo execute code within a given scope, this method updates the interpreter's\n`environment` field, visits all of the statements, and then restores the\nprevious value. As is always good practice in Java, it restores the previous\nenvironment using a finally clause. That way it gets restored even if an\nexception is thrown.\n\n\n\nSurprisingly, that's all we need to do in order to fully support local\nvariables, nesting, and shadowing. Go ahead and try this out:\n\n```lox\nvar a = \"global a\";\nvar b = \"global b\";\nvar c = \"global c\";\n{\n var a = \"outer a\";\n var b = \"outer b\";\n {\n var a = \"inner a\";\n print a;\n print b;\n print c;\n }\n print a;\n print b;\n print c;\n}\nprint a;\nprint b;\nprint c;\n```\n\nOur little interpreter can remember things now. We are inching closer to\nsomething resembling a full-featured programming language.\n\n
\n\n## Challenges\n\n1. The REPL no longer supports entering a single expression and automatically\n printing its result value. That's a drag. Add support to the REPL to let\n users type in both statements and expressions. If they enter a statement,\n execute it. If they enter an expression, evaluate it and display the result\n value.\n\n2. Maybe you want Lox to be a little more explicit about variable\n initialization. Instead of implicitly initializing variables to `nil`, make\n it a runtime error to access a variable that has not been initialized or\n assigned to, as in:\n\n ```lox\n // No initializers.\n var a;\n var b;\n\n a = \"assigned\";\n print a; // OK, was assigned first.\n\n print b; // Error!\n ```\n\n3. What does the following program do?\n\n ```lox\n var a = 1;\n {\n var a = a + 2;\n print a;\n }\n ```\n\n What did you *expect* it to do? Is it what you think it should do? What\n does analogous code in other languages you are familiar with do? What do\n you think users will expect this to do?\n\n
\n\n
\n\n## Design Note: Implicit Variable Declaration\n\nLox has distinct syntax for declaring a new variable and assigning to an\nexisting one. Some languages collapse those to only assignment syntax. Assigning\nto a non-existent variable automatically brings it into being. This is called\n**implicit variable declaration** and exists in Python, Ruby, and CoffeeScript,\namong others. JavaScript has an explicit syntax to declare variables, but can\nalso create new variables on assignment. Visual Basic has [an option to enable\nor disable implicit variables][vb].\n\n[vb]: https://msdn.microsoft.com/en-us/library/xe53dz5w(v=vs.100).aspx\n\nWhen the same syntax can assign or create a variable, each language must decide\nwhat happens when it isn't clear about which behavior the user intends. In\nparticular, each language must choose how implicit declaration interacts with\nshadowing, and which scope an implicitly declared variable goes into.\n\n* In Python, assignment always creates a variable in the current function's\n scope, even if there is a variable with the same name declared outside of\n the function.\n\n* Ruby avoids some ambiguity by having different naming rules for local and\n global variables. However, blocks in Ruby (which are more like closures than\n like \"blocks\" in C) have their own scope, so it still has the problem.\n Assignment in Ruby assigns to an existing variable outside of the current\n block if there is one with the same name. Otherwise, it creates a new\n variable in the current block's scope.\n\n* CoffeeScript, which takes after Ruby in many ways, is similar. It explicitly\n disallows shadowing by saying that assignment always assigns to a variable\n in an outer scope if there is one, all the way up to the outermost global\n scope. Otherwise, it creates the variable in the current function scope.\n\n* In JavaScript, assignment modifies an existing variable in any enclosing\n scope, if found. If not, it implicitly creates a new variable in the\n *global* scope.\n\nThe main advantage to implicit declaration is simplicity. There's less syntax\nand no \"declaration\" concept to learn. Users can just start assigning stuff and\nthe language figures it out.\n\nOlder, statically typed languages like C benefit from explicit declaration\nbecause they give the user a place to tell the compiler what type each variable\nhas and how much storage to allocate for it. In a dynamically typed,\ngarbage-collected language, that isn't really necessary, so you can get away\nwith making declarations implicit. It feels a little more \"scripty\", more \"you\nknow what I mean\".\n\nBut is that a good idea? Implicit declaration has some problems.\n\n* A user may intend to assign to an existing variable, but may have misspelled\n it. The interpreter doesn't know that, so it goes ahead and silently creates\n some new variable and the variable the user wanted to assign to still has\n its old value. This is particularly heinous in JavaScript where a typo will\n create a *global* variable, which may in turn interfere with other code.\n\n* JS, Ruby, and CoffeeScript use the presence of an existing variable with the\n same name -- even in an outer scope -- to determine whether or not an\n assignment creates a new variable or assigns to an existing one. That means\n adding a new variable in a surrounding scope can change the meaning of\n existing code. What was once a local variable may silently turn into an\n assignment to that new outer variable.\n\n* In Python, you may *want* to assign to some variable outside of the current\n function instead of creating a new variable in the current one, but you\n can't.\n\nOver time, the languages I know with implicit variable declaration ended up\nadding more features and complexity to deal with these problems.\n\n* Implicit declaration of global variables in JavaScript is universally\n considered a mistake today. \"Strict mode\" disables it and makes it a compile\n error.\n\n* Python added a `global` statement to let you explicitly assign to a global\n variable from within a function. Later, as functional programming and nested\n functions became more popular, they added a similar `nonlocal` statement to\n assign to variables in enclosing functions.\n\n* Ruby extended its block syntax to allow declaring certain variables to be\n explicitly local to the block even if the same name exists in an outer\n scope.\n\nGiven those, I think the simplicity argument is mostly lost. There is an\nargument that implicit declaration is the right *default* but I personally find\nthat less compelling.\n\nMy opinion is that implicit declaration made sense in years past when most\nscripting languages were heavily imperative and code was pretty flat. As\nprogrammers have gotten more comfortable with deep nesting, functional\nprogramming, and closures, it's become much more common to want access to\nvariables in outer scopes. That makes it more likely that users will run into\nthe tricky cases where it's not clear whether they intend their assignment to\ncreate a new variable or reuse a surrounding one.\n\nSo I prefer explicitly declaring variables, which is why Lox requires it.\n\n
\n" }, { "path": "book/strings.md", "content": "> \"Ah? A small aversion to menial labor?\" The doctor cocked an eyebrow.\n> \"Understandable, but misplaced. One should treasure those hum-drum\n> tasks that keep the body occupied but leave the mind and heart unfettered.\"\n>\n> Tad Williams, The Dragonbone Chair\n\nOur little VM can represent three types of values right now: numbers, Booleans,\nand `nil`. Those types have two important things in common: they're immutable\nand they're small. Numbers are the largest, and they still fit into two 64-bit\nwords. That's a small enough price that we can afford to pay it for all values,\neven Booleans and nils which don't need that much space.\n\nStrings, unfortunately, are not so petite. There's no maximum length for a\nstring. Even if we were to artificially cap it at some contrived limit like\n255 characters, that's still too much memory to spend\non every single value.\n\n\n\nWe need a way to support values whose sizes vary, sometimes greatly. This is\nexactly what dynamic allocation on the heap is designed for. We can allocate as\nmany bytes as we need. We get back a pointer that we'll use to keep track of the\nvalue as it flows through the VM.\n\n## Values and Objects\n\nUsing the heap for larger, variable-sized values and the stack for smaller,\natomic ones leads to a two-level representation. Every Lox value that you can\nstore in a variable or return from an expression will be a Value. For small,\nfixed-size types like numbers, the payload is stored directly inside the Value\nstruct itself.\n\nIf the object is larger, its data lives on the heap. Then the Value's payload is\na *pointer* to that blob of memory. We'll eventually have a handful of\nheap-allocated types in clox: strings, instances, functions, you get the idea.\nEach type has its own unique data, but there is also state they all share that\n[our future garbage collector][gc] will use to manage their memory.\n\n\"Field\n\n[gc]: garbage-collection.html\n\nWe'll call this common representation \"Obj\". Each Lox\nvalue whose state lives on the heap is an Obj. We can thus use a single new\nValueType case to refer to all heap-allocated types.\n\n\n\n^code val-obj (1 before, 1 after)\n\nWhen a Value's type is `VAL_OBJ`, the payload is a pointer to the heap memory,\nso we add another case to the union for that.\n\n^code union-object (1 before, 1 after)\n\nAs we did with the other value types, we crank out a couple of helpful macros\nfor working with Obj values.\n\n^code is-obj (1 before, 2 after)\n\nThis evaluates to `true` if the given Value is an Obj. If so, we can use this:\n\n^code as-obj (2 before, 1 after)\n\nIt extracts the Obj pointer from the value. We can also go the other way.\n\n^code obj-val (1 before, 2 after)\n\nThis takes a bare Obj pointer and wraps it in a full Value.\n\n## Struct Inheritance\n\nEvery heap-allocated value is an Obj, but Objs are\nnot all the same. For strings, we need the array of characters. When we get to\ninstances, they will need their data fields. A function object will need its\nchunk of bytecode. How do we handle different payloads and sizes? We can't use\nanother union like we did for Value since the sizes are all over the place.\n\n\n\nInstead, we'll use another technique. It's been around for ages, to the point\nthat the C specification carves out specific support for it, but I don't know\nthat it has a canonical name. It's an example of [*type punning*][pun], but that\nterm is too broad. In the absence of any better ideas, I'll call it **struct\ninheritance**, because it relies on structs and roughly follows how\nsingle-inheritance of state works in object-oriented languages.\n\n[pun]: https://en.wikipedia.org/wiki/Type_punning\n\nLike a tagged union, each Obj starts with a tag field that identifies what kind\nof object it is -- string, instance, etc. Following that are the payload fields.\nInstead of a union with cases for each type, each type is its own separate\nstruct. The tricky part is how to treat these structs uniformly since C has no\nconcept of inheritance or polymorphism. I'll explain that soon, but first lets\nget the preliminary stuff out of the way.\n\nThe name \"Obj\" itself refers to a struct that contains the state shared across\nall object types. It's sort of like the \"base class\" for objects. Because of\nsome cyclic dependencies between values and objects, we forward-declare it in\nthe \"value\" module.\n\n^code forward-declare-obj (2 before, 1 after)\n\nAnd the actual definition is in a new module.\n\n^code object-h\n\nRight now, it contains only the type tag. Shortly, we'll add some other\nbookkeeping information for memory management. The type enum is this:\n\n^code obj-type (1 before, 2 after)\n\nObviously, that will be more useful in later chapters after we add more\nheap-allocated types. Since we'll be accessing these tag types frequently, it's\nworth making a little macro that extracts the object type tag from a given\nValue.\n\n^code obj-type-macro (1 before, 2 after)\n\nThat's our foundation.\n\nNow, let's build strings on top of it. The payload for strings is defined in a\nseparate struct. Again, we need to forward-declare it.\n\n^code forward-declare-obj-string (1 before, 2 after)\n\nThe definition lives alongside Obj.\n\n^code obj-string (1 before, 2 after)\n\nA string object contains an array of characters. Those are stored in a separate,\nheap-allocated array so that we set aside only as much room as needed for each\nstring. We also store the number of bytes in the array. This isn't strictly\nnecessary but lets us tell how much memory is allocated for the string without\nwalking the character array to find the null terminator.\n\nBecause ObjString is an Obj, it also needs the state all Objs share. It\naccomplishes that by having its first field be an Obj. C specifies that struct\nfields are arranged in memory in the order that they are declared. Also, when\nyou nest structs, the inner struct's fields are expanded right in place. So the\nmemory for Obj and for ObjString looks like this:\n\n\"The\n\nNote how the first bytes of ObjString exactly line up with Obj. This is not a\ncoincidence -- C mandates it. This is designed to\nenable a clever pattern: You can take a pointer to a struct and safely convert\nit to a pointer to its first field and back.\n\n\n\nGiven an `ObjString*`, you can safely cast it to `Obj*` and then access the\n`type` field from it. Every ObjString \"is\" an Obj in the OOP sense of \"is\". When\nwe later add other object types, each struct will have an Obj as its first\nfield. Any code that wants to work with all objects can treat them as base\n`Obj*` and ignore any other fields that may happen to follow.\n\nYou can go in the other direction too. Given an `Obj*`, you can \"downcast\" it to\nan `ObjString*`. Of course, you need to ensure that the `Obj*` pointer you have\ndoes point to the `obj` field of an actual ObjString. Otherwise, you are\nunsafely reinterpreting random bits of memory. To detect that such a cast is\nsafe, we add another macro.\n\n^code is-string (1 before, 2 after)\n\nIt takes a Value, not a raw `Obj*` because most code in the VM works with\nValues. It relies on this inline function:\n\n^code is-obj-type (2 before, 2 after)\n\nPop quiz: Why not just put the body of this function right in the macro? What's\ndifferent about this one compared to the others? Right, it's because the body\nuses `value` twice. A macro is expanded by inserting the argument *expression*\nevery place the parameter name appears in the body. If a macro uses a parameter\nmore than once, that expression gets evaluated multiple times.\n\nThat's bad if the expression has side effects. If we put the body of\n`isObjType()` into the macro definition and then you did, say,\n\n```c\nIS_STRING(POP())\n```\n\nthen it would pop two values off the stack! Using a function fixes that.\n\nAs long as we ensure that we set the type tag correctly whenever we create an\nObj of some type, this macro will tell us when it's safe to cast a value to a\nspecific object type. We can do that using these:\n\n^code as-string (1 before, 2 after)\n\nThese two macros take a Value that is expected to contain a pointer to a valid\nObjString on the heap. The first one returns the `ObjString*` pointer. The\nsecond one steps through that to return the character array itself, since that's\noften what we'll end up needing.\n\n## Strings\n\nOK, our VM can now represent string values. It's time to add strings to the\nlanguage itself. As usual, we begin in the front end. The lexer already\ntokenizes string literals, so it's the parser's turn.\n\n^code table-string (1 before, 1 after)\n\nWhen the parser hits a string token, it calls this parse function:\n\n^code parse-string\n\nThis takes the string's characters directly from the\nlexeme. The `+ 1` and `- 2` parts trim the leading and trailing quotation marks.\nIt then creates a string object, wraps it in a Value, and stuffs it into the\nconstant table.\n\n\n\nTo create the string, we use `copyString()`, which is declared in `object.h`.\n\n^code copy-string-h (2 before, 1 after)\n\nThe compiler module needs to include that.\n\n^code compiler-include-object (2 before, 1 after)\n\nOur \"object\" module gets an implementation file where we define the new\nfunction.\n\n^code object-c\n\nFirst, we allocate a new array on the heap, just big enough for the string's\ncharacters and the trailing terminator, using\nthis low-level macro that allocates an array with a given element type and\ncount:\n\n^code allocate (2 before, 1 after)\n\nOnce we have the array, we copy over the characters from the lexeme and\nterminate it.\n\n\n\nYou might wonder why the ObjString can't just point back to the original\ncharacters in the source string. Some ObjStrings will be created dynamically at\nruntime as a result of string operations like concatenation. Those strings\nobviously need to dynamically allocate memory for the characters, which means\nthe string needs to *free* that memory when it's no longer needed.\n\nIf we had an ObjString for a string literal, and tried to free its character\narray that pointed into the original source code string, bad things would\nhappen. So, for literals, we preemptively copy the characters over to the heap.\nThis way, every ObjString reliably owns its character array and can free it.\n\nThe real work of creating a string object happens in this function:\n\n^code allocate-string (2 before)\n\nIt creates a new ObjString on the heap and then initializes its fields. It's\nsort of like a constructor in an OOP language. As such, it first calls the \"base\nclass\" constructor to initialize the Obj state, using a new macro.\n\n^code allocate-obj (1 before, 2 after)\n\nLike the previous macro, this exists mainly to\navoid the need to redundantly cast a `void*` back to the desired type. The\nactual functionality is here:\n\n\n\n^code allocate-object (2 before, 2 after)\n\nIt allocates an object of the given size on the heap. Note that the size is\n*not* just the size of Obj itself. The caller passes in the number of bytes so\nthat there is room for the extra payload fields needed by the specific object\ntype being created.\n\nThen it initializes the Obj state -- right now, that's just the type tag. This\nfunction returns to `allocateString()`, which finishes initializing the ObjString\nfields. *Voilà*, we can compile and execute string\nliterals.\n\n\n\n## Operations on Strings\n\nOur fancy strings are there, but they don't do much of anything yet. A good\nfirst step is to make the existing print code not barf on the new value type.\n\n^code call-print-object (1 before, 1 after)\n\nIf the value is a heap-allocated object, it defers to a helper function over in\nthe \"object\" module.\n\n^code print-object-h (1 before, 2 after)\n\nThe implementation looks like this:\n\n^code print-object\n\nWe have only a single object type now, but this function will sprout additional\nswitch cases in later chapters. For string objects, it simply prints the character array as a C string.\n\n\n\nThe equality operators also need to gracefully handle strings. Consider:\n\n```lox\n\"string\" == \"string\"\n```\n\nThese are two separate string literals. The compiler will make two separate\ncalls to `copyString()`, create two distinct ObjString objects and store them as\ntwo constants in the chunk. They are different objects in the heap. But our\nusers (and thus we) expect strings to have value equality. The above expression\nshould evaluate to `true`. That requires a little special support.\n\n^code strings-equal (1 before, 1 after)\n\nIf the two values are both strings, then they are equal if their character\narrays contain the same characters, regardless of whether they are two separate\nobjects or the exact same one. This does mean that string equality is slower\nthan equality on other types since it has to walk the whole string. We'll revise\nthat [later][hash], but this gives us the right semantics for now.\n\n[hash]: hash-tables.html\n\nFinally, in order to use `memcmp()` and the new stuff in the \"object\" module, we\nneed a couple of includes. Here:\n\n^code value-include-string (1 before, 2 after)\n\nAnd here:\n\n^code value-include-object (2 before, 1 after)\n\n### Concatenation\n\nFull-grown languages provide lots of operations for working with strings --\naccess to individual characters, the string's length, changing case, splitting,\njoining, searching, etc. When you implement your language, you'll likely want\nall that. But for this book, we keep things *very* minimal.\n\nThe only interesting operation we support on strings is `+`. If you use that\noperator on two string objects, it produces a new string that's a concatenation\nof the two operands. Since Lox is dynamically typed, we can't tell which\nbehavior is needed at compile time because we don't know the types of the\noperands until runtime. Thus, the `OP_ADD` instruction dynamically inspects the\noperands and chooses the right operation.\n\n^code add-strings (1 before, 1 after)\n\nIf both operands are strings, it concatenates. If they're both numbers, it adds\nthem. Any other combination of operand types is a\nruntime error.\n\n\n\nTo concatenate strings, we define a new function.\n\n^code concatenate\n\nIt's pretty verbose, as C code that works with strings tends to be. First, we\ncalculate the length of the result string based on the lengths of the operands.\nWe allocate a character array for the result and then copy the two halves in. As\nalways, we carefully ensure the string is terminated.\n\nIn order to call `memcpy()`, the VM needs an include.\n\n^code vm-include-string (1 before, 2 after)\n\nFinally, we produce an ObjString to contain those characters. This time we use a\nnew function, `takeString()`.\n\n^code take-string-h (2 before, 1 after)\n\nThe implementation looks like this:\n\n^code take-string\n\nThe previous `copyString()` function assumes it *cannot* take ownership of the\ncharacters you pass in. Instead, it conservatively creates a copy of the\ncharacters on the heap that the ObjString can own. That's the right thing for\nstring literals where the passed-in characters are in the middle of the source\nstring.\n\nBut, for concatenation, we've already dynamically allocated a character array on\nthe heap. Making another copy of that would be redundant (and would mean\n`concatenate()` has to remember to free its copy). Instead, this function claims\nownership of the string you give it.\n\nAs usual, stitching this functionality together requires a couple of includes.\n\n^code vm-include-object-memory (1 before, 1 after)\n\n## Freeing Objects\n\nBehold this innocuous-seeming expression:\n\n```lox\n\"st\" + \"ri\" + \"ng\"\n```\n\nWhen the compiler chews through this, it allocates an ObjString for each of\nthose three string literals and stores them in the chunk's constant table and\ngenerates this bytecode:\n\n\n\n```text\n0000 OP_CONSTANT 0 \"st\"\n0002 OP_CONSTANT 1 \"ri\"\n0004 OP_ADD\n0005 OP_CONSTANT 2 \"ng\"\n0007 OP_ADD\n0008 OP_RETURN\n```\n\nThe first two instructions push `\"st\"` and `\"ri\"` onto the stack. Then the\n`OP_ADD` pops those and concatenates them. That dynamically allocates a new\n`\"stri\"` string on the heap. The VM pushes that and then pushes the `\"ng\"`\nconstant. The last `OP_ADD` pops `\"stri\"` and `\"ng\"`, concatenates them, and\npushes the result: `\"string\"`. Great, that's what we expect.\n\nBut, wait. What happened to that `\"stri\"` string? We dynamically allocated it,\nthen the VM discarded it after concatenating it with `\"ng\"`. We popped it from\nthe stack and no longer have a reference to it, but we never freed its memory.\nWe've got ourselves a classic memory leak.\n\nOf course, it's perfectly fine for the *Lox program* to forget about\nintermediate strings and not worry about freeing them. Lox automatically manages\nmemory on the user's behalf. The responsibility to manage memory doesn't\n*disappear*. Instead, it falls on our shoulders as VM implementers.\n\nThe full solution is a [garbage collector][gc] that\nreclaims unused memory while the program is running. We've got some other stuff\nto get in place before we're ready to tackle that project. Until then, we are\nliving on borrowed time. The longer we wait to add the collector, the harder it\nis to do.\n\n\n\nToday, we should at least do the bare minimum: avoid *leaking* memory by making\nsure the VM can still find every allocated object even if the Lox program itself\nno longer references them. There are many sophisticated techniques that advanced\nmemory managers use to allocate and track memory for objects. We're going to\ntake the simplest practical approach.\n\nWe'll create a linked list that stores every Obj. The VM can traverse that\nlist to find every single object that has been allocated on the heap, whether or\nnot the user's program or the VM's stack still has a reference to it.\n\nWe could define a separate linked list node struct but then we'd have to\nallocate those too. Instead, we'll use an **intrusive list** -- the Obj struct\nitself will be the linked list node. Each Obj gets a pointer to the next Obj in\nthe chain.\n\n^code next-field (2 before, 1 after)\n\nThe VM stores a pointer to the head of the list.\n\n^code objects-root (1 before, 1 after)\n\nWhen we first initialize the VM, there are no allocated objects.\n\n^code init-objects-root (1 before, 1 after)\n\nEvery time we allocate an Obj, we insert it in the list.\n\n^code add-to-list (1 before, 1 after)\n\nSince this is a singly linked list, the easiest place to insert it is as the\nhead. That way, we don't need to also store a pointer to the tail and keep it\nupdated.\n\nThe \"object\" module is directly using the global `vm` variable from the \"vm\"\nmodule, so we need to expose that externally.\n\n^code extern-vm (2 before, 1 after)\n\nEventually, the garbage collector will free memory while the VM is still\nrunning. But, even then, there will usually be unused objects still lingering in\nmemory when the user's program completes. The VM should free those too.\n\nThere's no sophisticated logic for that. Once the program is done, we can free\n*every* object. We can and should implement that now.\n\n^code call-free-objects (1 before, 1 after)\n\nThat empty function we defined [way back when][vm] finally does something! It\ncalls this:\n\n[vm]: a-virtual-machine.html#an-instruction-execution-machine\n\n^code free-objects-h (1 before, 2 after)\n\nHere's how we free the objects:\n\n^code free-objects\n\nThis is a CS 101 textbook implementation of walking a linked list and freeing\nits nodes. For each node, we call:\n\n^code free-object\n\nWe aren't only freeing the Obj itself. Since some object types also allocate\nother memory that they own, we also need a little type-specific code to handle\neach object type's special needs. Here, that means we free the character array\nand then free the ObjString. Those both use one last memory management macro.\n\n^code free (1 before, 2 after)\n\nIt's a tiny wrapper around `reallocate()` that\n\"resizes\" an allocation down to zero bytes.\n\n\n\nAs usual, we need an include to wire everything together.\n\n^code memory-include-object (1 before, 2 after)\n\nThen in the implementation file:\n\n^code memory-include-vm (1 before, 2 after)\n\nWith this, our VM no longer leaks memory. Like a good C program, it cleans up\nits mess before exiting. But it doesn't free any objects while the VM is\nrunning. Later, when it's possible to write longer-running Lox programs, the VM\nwill eat more and more memory as it goes, not relinquishing a single byte until\nthe entire program is done.\n\nWe won't address that until we've added [a real garbage collector][gc], but this\nis a big step. We now have the infrastructure to support a variety of different\nkinds of dynamically allocated objects. And we've used that to add strings to\nclox, one of the most used types in most programming languages. Strings in turn\nenable us to build another fundamental data type, especially in dynamic\nlanguages: the venerable [hash table][]. But that's for the next chapter...\n\n[hash table]: hash-tables.html\n\n
\n\n## Challenges\n\n1. Each string requires two separate dynamic allocations -- one for the\n ObjString and a second for the character array. Accessing the characters\n from a value requires two pointer indirections, which can be bad for\n performance. A more efficient solution relies on a technique called\n **[flexible array members][]**. Use that to store the ObjString and its\n character array in a single contiguous allocation.\n\n2. When we create the ObjString for each string literal, we copy the characters\n onto the heap. That way, when the string is later freed, we know it is safe\n to free the characters too.\n\n This is a simpler approach but wastes some memory, which might be a problem\n on very constrained devices. Instead, we could keep track of which\n ObjStrings own their character array and which are \"constant strings\" that\n just point back to the original source string or some other non-freeable\n location. Add support for this.\n\n3. If Lox was your language, what would you have it do when a user tries to use\n `+` with one string operand and the other some other type? Justify your\n choice. What do other languages do?\n\n[flexible array members]: https://en.wikipedia.org/wiki/Flexible_array_member\n\n
\n\n
\n\n## Design Note: String Encoding\n\nIn this book, I try not to shy away from the gnarly problems you'll run into in\na real language implementation. We might not always use the most *sophisticated*\nsolution -- it's an intro book after all -- but I don't think it's honest to\npretend the problem doesn't exist at all. However, I did skirt around one really\nnasty conundrum: deciding how to represent strings.\n\nThere are two facets to a string encoding:\n\n* **What is a single \"character\" in a string?** How many different values are\n there and what do they represent? The first widely adopted standard answer\n to this was [ASCII][]. It gave you 127 different character values and\n specified what they were. It was great... if you only ever cared about\n English. While it has weird, mostly forgotten characters like \"record\n separator\" and \"synchronous idle\", it doesn't have a single umlaut, acute,\n or grave. It can't represent \"jalapeño\", \"naïve\", \"Gruyère\", or \"Mötley Crüe\".\n\n \n\n Next came [Unicode][]. Initially, it supported 16,384 different characters\n (**code points**), which fit nicely in 16 bits with a couple of bits to\n spare. Later that grew and grew, and now there are well over 100,000\n different code points including such vital instruments of human\n communication as 💩 (Unicode Character 'PILE OF POO', `U+1F4A9`).\n\n Even that long list of code points is not enough to represent each possible\n visible glyph a language might support. To handle that, Unicode also has\n **combining characters** that modify a preceding code point. For example,\n \"a\" followed by the combining character \"¨\" gives you \"ä\". (To make things\n more confusing Unicode *also* has a single code point that looks like \"ä\".)\n\n If a user accesses the fourth \"character\" in \"naïve\", do they expect to get\n back \"v\" or “¨”? The former means they are thinking of each code\n point and its combining character as a single unit -- what Unicode calls an\n **extended grapheme cluster** -- the latter means they are thinking in\n individual code points. Which do your users expect?\n\n* **How is a single unit represented in memory?** Most systems using ASCII\n gave a single byte to each character and left the high bit unused. Unicode\n has a handful of common encodings. UTF-16 packs most code points into 16\n bits. That was great when every code point fit in that size. When that\n overflowed, they added *surrogate pairs* that use multiple 16-bit code units\n to represent a single code point. UTF-32 is the next evolution of\n UTF-16 -- it gives a full 32 bits to each and every code point.\n\n UTF-8 is more complex than either of those. It uses a variable number of\n bytes to encode a code point. Lower-valued code points fit in fewer bytes.\n Since each character may occupy a different number of bytes, you can't\n directly index into the string to find a specific code point. If you want,\n say, the 10th code point, you don't know how many bytes into the string that\n is without walking and decoding all of the preceding ones.\n\n[ascii]: https://en.wikipedia.org/wiki/ASCII\n[unicode]: https://en.wikipedia.org/wiki/Unicode\n\nChoosing a character representation and encoding involves fundamental\ntrade-offs. Like many things in engineering, there's no perfect solution:\n\n\n\n* ASCII is memory efficient and fast, but it kicks non-Latin languages to the\n side.\n\n* UTF-32 is fast and supports the whole Unicode range, but wastes a lot of\n memory given that most code points do tend to be in the lower range of\n values, where a full 32 bits aren't needed.\n\n* UTF-8 is memory efficient and supports the whole Unicode range, but its\n variable-length encoding makes it slow to access arbitrary code points.\n\n* UTF-16 is worse than all of them -- an ugly consequence of Unicode\n outgrowing its earlier 16-bit range. It's less memory efficient than UTF-8\n but is still a variable-length encoding thanks to surrogate pairs. Avoid it\n if you can. Alas, if your language needs to run on or interoperate with the\n browser, the JVM, or the CLR, you might be stuck with it, since those all\n use UTF-16 for their strings and you don't want to have to convert every\n time you pass a string to the underlying system.\n\nOne option is to take the maximal approach and do the \"rightest\" thing. Support\nall the Unicode code points. Internally, select an encoding for each string\nbased on its contents -- use ASCII if every code point fits in a byte, UTF-16 if\nthere are no surrogate pairs, etc. Provide APIs to let users iterate over both\ncode points and extended grapheme clusters.\n\nThis covers all your bases but is really complex. It's a lot to implement,\ndebug, and optimize. When serializing strings or interoperating with other\nsystems, you have to deal with all of the encodings. Users need to understand\nthe two indexing APIs and know which to use when. This is the approach that\nnewer, big languages tend to take -- like Raku and Swift.\n\nA simpler compromise is to always encode using UTF-8 and only expose an API that\nworks with code points. For users that want to work with grapheme clusters, let\nthem use a third-party library for that. This is less Latin-centric than ASCII\nbut not much more complex. You lose fast direct indexing by code point, but you\ncan usually live without that or afford to make it *O(n)* instead of *O(1)*.\n\nIf I were designing a big workhorse language for people writing large\napplications, I'd probably go with the maximal approach. For my little embedded\nscripting language [Wren][], I went with UTF-8 and code points.\n\n[wren]: http://wren.io\n\n
\n" }, { "path": "book/superclasses.md", "content": "> You can choose your friends but you sho' can't choose your family, an' they're\n> still kin to you no matter whether you acknowledge ’em or not, and it\n> makes you look right silly when you don't.\n>\n> Harper Lee, To Kill a Mockingbird\n\nThis is the very last chapter where we add new functionality to our VM. We've\npacked almost the entire Lox language in there already. All that remains is\ninheriting methods and calling superclass methods. We have [another\nchapter][optimization] after this one, but it introduces no new behavior. It\nonly makes existing stuff faster. Make it to the end\nof this one, and you'll have a complete Lox implementation.\n\n\n\n[optimization]: optimization.html\n\nSome of the material in this chapter will remind you of jlox. The way we resolve\nsuper calls is pretty much the same, though viewed through clox's more complex\nmechanism for storing state on the stack. But we have an entirely different,\nmuch faster, way of handling inherited method calls this time around.\n\n## Inheriting Methods\n\nWe'll kick things off with method inheritance since it's the simpler piece. To\nrefresh your memory, Lox inheritance syntax looks like this:\n\n```lox\nclass Doughnut {\n cook() {\n print \"Dunk in the fryer.\";\n }\n}\n\nclass Cruller < Doughnut {\n finish() {\n print \"Glaze with icing.\";\n }\n}\n```\n\nHere, the Cruller class inherits from Doughnut and thus, instances of Cruller\ninherit the `cook()` method. I don't know why I'm belaboring this. You know how\ninheritance works. Let's start compiling the new syntax.\n\n^code compile-superclass (2 before, 1 after)\n\nAfter we compile the class name, if the next token is a `<`, then we found a\nsuperclass clause. We consume the superclass's identifier token, then call\n`variable()`. That function takes the previously consumed token, treats it as a\nvariable reference, and emits code to load the variable's value. In other words,\nit looks up the superclass by name and pushes it onto the stack.\n\nAfter that, we call `namedVariable()` to load the subclass doing the inheriting\nonto the stack, followed by an `OP_INHERIT` instruction. That instruction\nwires up the superclass to the new subclass. In the last chapter, we defined an\n`OP_METHOD` instruction to mutate an existing class object by adding a method to\nits method table. This is similar -- the `OP_INHERIT` instruction takes an\nexisting class and applies the effect of inheritance to it.\n\nIn the previous example, when the compiler works through this bit of syntax:\n\n```lox\nclass Cruller < Doughnut {\n```\n\nThe result is this bytecode:\n\n\"The\n\nBefore we implement the new `OP_INHERIT` instruction, we have an edge case to\ndetect.\n\n^code inherit-self (1 before, 1 after)\n\nA class cannot be its own superclass. Unless you have\naccess to a deranged nuclear physicist and a very heavily modified DeLorean, you\ncannot inherit from yourself.\n\n\n\n### Executing inheritance\n\nNow onto the new instruction.\n\n^code inherit-op (1 before, 1 after)\n\nThere are no operands to worry about. The two values we need -- superclass and\nsubclass -- are both found on the stack. That means disassembling is easy.\n\n^code disassemble-inherit (1 before, 1 after)\n\nThe interpreter is where the action happens.\n\n^code interpret-inherit (1 before, 1 after)\n\nFrom the top of the stack down, we have the subclass then the superclass. We\ngrab both of those and then do the inherit-y bit. This is where clox takes a\ndifferent path than jlox. In our first interpreter, each subclass stored a\nreference to its superclass. On method access, if we didn't find the method in\nthe subclass's method table, we recursed through the inheritance chain looking\nat each ancestor's method table until we found it.\n\nFor example, calling `cook()` on an instance of Cruller sends jlox on this\njourney:\n\n\"Resolving\n\nThat's a lot of work to perform during method *invocation* time. It's slow, and\nworse, the farther an inherited method is up the ancestor chain, the slower it\ngets. Not a great performance story.\n\nThe new approach is much faster. When the subclass is declared, we copy all of\nthe inherited class's methods down into the subclass's own method table. Later,\nwhen *calling* a method, any method inherited from a superclass will be found\nright in the subclass's own method table. There is no extra runtime work needed\nfor inheritance at all. By the time the class is declared, the work is done.\nThis means inherited method calls are exactly as fast as normal method calls --\na single hash table lookup.\n\n\"Resolving\n\n\n\nI've sometimes heard this technique called \"copy-down inheritance\". It's simple\nand fast, but, like most optimizations, you get to use it only under certain\nconstraints. It works in Lox because Lox classes are *closed*. Once a class\ndeclaration is finished executing, the set of methods for that class can never\nchange.\n\nIn languages like Ruby, Python, and JavaScript, it's possible to crack open an existing class and jam some new methods into\nit or even remove them. That would break our optimization because if those\nmodifications happened to a superclass *after* the subclass declaration\nexecuted, the subclass would not pick up those changes. That breaks a user's\nexpectation that inheritance always reflects the current state of the\nsuperclass.\n\n\n\nFortunately for us (but not for users who like the feature, I guess), Lox\ndoesn't let you patch monkeys or punch ducks, so we can safely apply this\noptimization.\n\nWhat about method overrides? Won't copying the superclass's methods into the\nsubclass's method table clash with the subclass's own methods? Fortunately, no.\nWe emit the `OP_INHERIT` after the `OP_CLASS` instruction that creates the\nsubclass but before any method declarations and `OP_METHOD` instructions have\nbeen compiled. At the point that we copy the superclass's methods down, the\nsubclass's method table is empty. Any methods the subclass overrides will\noverwrite those inherited entries in the table.\n\n### Invalid superclasses\n\nOur implementation is simple and fast, which is just the way I like my VM code.\nBut it's not robust. Nothing prevents a user from inheriting from an object that\nisn't a class at all:\n\n```lox\nvar NotClass = \"So not a class\";\nclass OhNo < NotClass {}\n```\n\nObviously, no self-respecting programmer would write that, but we have to guard\nagainst potential Lox users who have no self respect. A simple runtime check\nfixes that.\n\n^code inherit-non-class (1 before, 1 after)\n\nIf the value we loaded from the identifier in the superclass clause isn't an\nObjClass, we report a runtime error to let the user know what we think of them\nand their code.\n\n## Storing Superclasses\n\nDid you notice that when we added method inheritance, we didn't actually add any\nreference from a subclass to its superclass? After we copy the inherited methods\nover, we forget the superclass entirely. We don't need to keep a handle on the\nsuperclass, so we don't.\n\nThat won't be sufficient to support super calls. Since a subclass may override the superclass method, we need to be able to get\nour hands on superclass method tables. Before we get to that mechanism, I want \nto refresh your memory on how super calls are statically resolved.\n\n\n\nBack in the halcyon days of jlox, I showed you [this tricky example][example] to\nexplain the way super calls are dispatched:\n\n[example]: inheritance.html#semantics\n\n```lox\nclass A {\n method() {\n print \"A method\";\n }\n}\n\nclass B < A {\n method() {\n print \"B method\";\n }\n\n test() {\n super.method();\n }\n}\n\nclass C < B {}\n\nC().test();\n```\n\nInside the body of the `test()` method, `this` is an instance of C. If super\ncalls were resolved relative to the superclass of the *receiver*, then we would\nlook in C's superclass, B. But super calls are resolved relative to the\nsuperclass of the *surrounding class where the super call occurs*. In this case,\nwe are in B's `test()` method, so the superclass is A, and the program should\nprint \"A method\".\n\nThis means that super calls are not resolved dynamically based on the runtime\ninstance. The superclass used to look up the method is a static -- practically\nlexical -- property of where the call occurs. When we added inheritance to jlox,\nwe took advantage of that static aspect by storing the superclass in the same\nEnvironment structure we used for all lexical scopes. Almost as if the\ninterpreter saw the above program like this:\n\n```lox\nclass A {\n method() {\n print \"A method\";\n }\n}\n\nvar Bs_super = A;\nclass B < A {\n method() {\n print \"B method\";\n }\n\n test() {\n runtimeSuperCall(Bs_super, \"method\");\n }\n}\n\nvar Cs_super = B;\nclass C < B {}\n\nC().test();\n```\n\nEach subclass has a hidden variable storing a reference to its superclass.\nWhenever we need to perform a super call, we access the superclass from that\nvariable and tell the runtime to start looking for methods there.\n\nWe'll take the same path with clox. The difference is that instead of jlox's\nheap-allocated Environment class, we have the bytecode VM's value stack and\nupvalue system. The machinery is a little different, but the overall effect is\nthe same.\n\n### A superclass local variable\n\nOur compiler already emits code to load the superclass onto the stack. Instead\nof leaving that slot as a temporary, we create a new scope and make it a local\nvariable.\n\n^code superclass-variable (2 before, 2 after)\n\nCreating a new lexical scope ensures that if we declare two classes in the same\nscope, each has a different local slot to store its superclass. Since we always\nname this variable \"super\", if we didn't make a scope for each subclass, the\nvariables would collide.\n\nWe name the variable \"super\" for the same reason we use \"this\" as the name of\nthe hidden local variable that `this` expressions resolve to: \"super\" is a\nreserved word, which guarantees the compiler's hidden variable won't collide\nwith a user-defined one.\n\nThe difference is that when compiling `this` expressions, we conveniently have a\ntoken sitting around whose lexeme is \"this\". We aren't so lucky here. Instead,\nwe add a little helper function to create a synthetic token for the given constant string.\n\n^code synthetic-token\n\n\n\nSince we opened a local scope for the superclass variable, we need to close it.\n\n^code end-superclass-scope (1 before, 2 after)\n\nWe pop the scope and discard the \"super\" variable after compiling the class body\nand its methods. That way, the variable is accessible in all of the methods of\nthe subclass. It's a somewhat pointless optimization, but we create the scope\nonly if there *is* a superclass clause. Thus we need to close the scope only if\nthere is one.\n\nTo track that, we could declare a little local variable in `classDeclaration()`.\nBut soon, other functions in the compiler will need to know whether the\nsurrounding class is a subclass or not. So we may as well give our future selves\na hand and store this fact as a field in the ClassCompiler now.\n\n^code has-superclass (2 before, 1 after)\n\nWhen we first initialize a ClassCompiler, we assume it is not a subclass.\n\n^code init-has-superclass (1 before, 1 after)\n\nThen, if we see a superclass clause, we know we are compiling a subclass.\n\n^code set-has-superclass (1 before, 1 after)\n\nThis machinery gives us a mechanism at runtime to access the superclass object\nof the surrounding subclass from within any of the subclass's methods -- simply\nemit code to load the variable named \"super\". That variable is a local outside\nof the method body, but our existing upvalue support enables the VM to capture\nthat local inside the body of the method or even in functions nested inside that\nmethod.\n\n## Super Calls\n\nWith that runtime support in place, we are ready to implement super calls. As\nusual, we go front to back, starting with the new syntax. A super call begins, naturally enough, with the `super` keyword.\n\n\n\n^code table-super (1 before, 1 after)\n\nWhen the expression parser lands on a `super` token, control jumps to a new\nparsing function which starts off like so:\n\n^code super\n\nThis is pretty different from how we compiled `this` expressions. Unlike `this`,\na `super` token is not a standalone expression.\nInstead, the dot and method name following it are inseparable parts of the\nsyntax. However, the parenthesized argument list is separate. As with normal\nmethod access, Lox supports getting a reference to a superclass method as a\nclosure without invoking it:\n\n\n\n```lox\nclass A {\n method() {\n print \"A\";\n }\n}\n\nclass B < A {\n method() {\n var closure = super.method;\n closure(); // Prints \"A\".\n }\n}\n```\n\nIn other words, Lox doesn't really have super *call* expressions, it has super\n*access* expressions, which you can choose to immediately invoke if you want. So\nwhen the compiler hits a `super` token, we consume the subsequent `.` token and\nthen look for a method name. Methods are looked up dynamically, so we use\n`identifierConstant()` to take the lexeme of the method name token and store it\nin the constant table just like we do for property access expressions.\n\nHere is what the compiler does after consuming those tokens:\n\n^code super-get (1 before, 1 after)\n\nIn order to access a *superclass method* on *the current instance*, the runtime\nneeds both the receiver *and* the superclass of the surrounding method's class.\nThe first `namedVariable()` call generates code to look up the current receiver\nstored in the hidden variable \"this\" and push it onto the stack. The second\n`namedVariable()` call emits code to look up the superclass from its \"super\"\nvariable and push that on top.\n\nFinally, we emit a new `OP_GET_SUPER` instruction with an operand for the\nconstant table index of the method name. That's a lot to hold in your head. To\nmake it tangible, consider this example program:\n\n```lox\nclass Doughnut {\n cook() {\n print \"Dunk in the fryer.\";\n this.finish(\"sprinkles\");\n }\n\n finish(ingredient) {\n print \"Finish with \" + ingredient;\n }\n}\n\nclass Cruller < Doughnut {\n finish(ingredient) {\n // No sprinkles, always icing.\n super.finish(\"icing\");\n }\n}\n```\n\nThe bytecode emitted for the `super.finish(\"icing\")` expression looks and works\nlike this:\n\n\"The\n\nThe first three instructions give the runtime access to the three pieces of\ninformation it needs to perform the super access:\n\n1. The first instruction loads **the instance** onto the stack.\n\n2. The second instruction loads **the superclass where the method is\n resolved**.\n\n3. Then the new `OP_GET_SUPER` instuction encodes **the name of the method to\n access** as an operand.\n\nThe remaining instructions are the normal bytecode for evaluating an argument\nlist and calling a function.\n\nWe're almost ready to implement the new `OP_GET_SUPER` instruction in the\ninterpreter. But before we do, the compiler has some errors it is responsible\nfor reporting.\n\n^code super-errors (1 before, 1 after)\n\nA super call is meaningful only inside the body of a method (or in a function\nnested inside a method), and only inside the method of a class that has a\nsuperclass. We detect both of these cases using the value of `currentClass`. If\nthat's `NULL` or points to a class with no superclass, we report those errors.\n\n### Executing super accesses\n\nAssuming the user didn't put a `super` expression where it's not allowed, their\ncode passes from the compiler over to the runtime. We've got ourselves a new\ninstruction.\n\n^code get-super-op (1 before, 1 after)\n\nWe disassemble it like other opcodes that take a constant table index operand.\n\n^code disassemble-get-super (1 before, 1 after)\n\nYou might anticipate something harder, but interpreting the new instruction is\nsimilar to executing a normal property access.\n\n^code interpret-get-super (1 before, 1 after)\n\nAs with properties, we read the method name from the\nconstant table. Then we pass that to `bindMethod()` which looks up the method in\nthe given class's method table and creates an ObjBoundMethod to bundle the\nresulting closure to the current instance.\n\nThe key difference is *which* class we pass to\n`bindMethod()`. With a normal property access, we use the ObjInstances's own\nclass, which gives us the dynamic dispatch we want. For a super call, we don't\nuse the instance's class. Instead, we use the statically resolved superclass of\nthe containing class, which the compiler has conveniently ensured is sitting on\ntop of the stack waiting for us.\n\nWe pop that superclass and pass it to `bindMethod()`, which correctly skips over\nany overriding methods in any of the subclasses between that superclass and the\ninstance's own class. It also correctly includes any methods inherited by the\nsuperclass from any of *its* superclasses.\n\nThe rest of the behavior is the same. Popping the superclass leaves the instance\nat the top of the stack. When `bindMethod()` succeeds, it pops the instance and\npushes the new bound method. Otherwise, it reports a runtime error and returns\n`false`. In that case, we abort the interpreter.\n\n\n\n### Faster super calls\n\nWe have superclass method accesses working now. And since the returned object is\nan ObjBoundMethod that you can then invoke, we've got super *calls* working too.\nJust like last chapter, we've reached a point where our VM has the complete,\ncorrect semantics.\n\nBut, also like last chapter, it's pretty slow. Again, we're heap allocating an\nObjBoundMethod for each super call even though most of the time the very next\ninstruction is an `OP_CALL` that immediately unpacks that bound method, invokes\nit, and then discards it. In fact, this is even more likely to be true for\nsuper calls than for regular method calls. At least with method calls there is\na chance that the user is actually invoking a function stored in a field. With\nsuper calls, you're *always* looking up a method. The only question is whether\nyou invoke it immediately or not.\n\nThe compiler can certainly answer that question for itself if it sees a left\nparenthesis after the superclass method name, so we'll go ahead and perform the\nsame optimization we did for method calls. Take out the two lines of code that\nload the superclass and emit `OP_GET_SUPER`, and replace them with this:\n\n^code super-invoke (1 before, 1 after)\n\nNow before we emit anything, we look for a parenthesized argument list. If we\nfind one, we compile that. Then we load the superclass. After that, we emit a\nnew `OP_SUPER_INVOKE` instruction. This superinstruction combines the behavior of\n`OP_GET_SUPER` and `OP_CALL`, so it takes two operands: the constant table index\nof the method name to look up and the number of arguments to pass to it.\n\n\n\nOtherwise, if we don't find a `(`, we continue to compile the expression as a\nsuper access like we did before and emit an `OP_GET_SUPER`.\n\nDrifting down the compilation pipeline, our first stop is a new instruction.\n\n^code super-invoke-op (1 before, 1 after)\n\nAnd just past that, its disassembler support.\n\n^code disassemble-super-invoke (1 before, 1 after)\n\nA super invocation instruction has the same set of operands as `OP_INVOKE`, so\nwe reuse the same helper to disassemble it. Finally, the pipeline dumps us into\nthe interpreter.\n\n^code interpret-super-invoke (2 before, 1 after)\n\nThis handful of code is basically our implementation of `OP_INVOKE` mixed\ntogether with a dash of `OP_GET_SUPER`. There are some differences in how the\nstack is organized, though. With an unoptimized super call, the superclass is\npopped and replaced by the ObjBoundMethod for the resolved function *before* the\narguments to the call are executed. This ensures that by the time the `OP_CALL`\nis executed, the bound method is *under* the argument list, where the runtime\nexpects it to be for a closure call.\n\nWith our optimized instructions, things are shuffled a bit:\n\n\"The\n\nNow resolving the superclass method is part of the *invocation*, so the\narguments need to already be on the stack at the point that we look up the\nmethod. This means the superclass object is on top of the arguments.\n\nAside from that, the behavior is roughly the same as an `OP_GET_SUPER` followed\nby an `OP_CALL`. First, we pull out the method name and argument count operands.\nThen we pop the superclass off the top of the stack so that we can look up the\nmethod in its method table. This conveniently leaves the stack set up just right\nfor a method call.\n\nWe pass the superclass, method name, and argument count to our existing\n`invokeFromClass()` function. That function looks up the given method on the\ngiven class and attempts to create a call to it with the given arity. If a\nmethod could not be found, it returns `false`, and we bail out of the\ninterpreter. Otherwise, `invokeFromClass()` pushes a new CallFrame onto the call\nstack for the method's closure. That invalidates the interpreter's cached\nCallFrame pointer, so we refresh `frame`.\n\n## A Complete Virtual Machine\n\nTake a look back at what we've created. By my count, we wrote around 2,500 lines\nof fairly clean, straightforward C. That little program contains a complete\nimplementation of the -- quite high-level! -- Lox language, with a whole\nprecedence table full of expression types and a suite of control flow\nstatements. We implemented variables, functions, closures, classes, fields,\nmethods, and inheritance.\n\nEven more impressive, our implementation is portable to any platform with a C\ncompiler, and is fast enough for real-world production use. We have a\nsingle-pass bytecode compiler, a tight virtual machine interpreter for our\ninternal instruction set, compact object representations, a stack for storing\nvariables without heap allocation, and a precise garbage collector.\n\nIf you go out and start poking around in the implementations of Lua, Python, or\nRuby, you will be surprised by how much of it now looks familiar to you. You\nhave seriously leveled up your knowledge of how programming languages work,\nwhich in turn gives you a deeper understanding of programming itself. It's like\nyou used to be a race car driver, and now you can pop the hood and repair the\nengine too.\n\nYou can stop here if you like. The two implementations of Lox you have are\ncomplete and full featured. You built the car and can drive it wherever you want\nnow. But if you are looking to have more fun tuning and tweaking for even\ngreater performance out on the track, there is one more chapter. We don't add\nany new capabilities, but we roll in a couple of classic optimizations to\nsqueeze even more perf out. If that sounds fun, [keep reading][opt]...\n\n[opt]: optimization.html\n\n
\n\n## Challenges\n\n1. A tenet of object-oriented programming is that a class should ensure new\n objects are in a valid state. In Lox, that means defining an initializer\n that populates the instance's fields. Inheritance complicates invariants\n because the instance must be in a valid state according to all of the\n classes in the object's inheritance chain.\n\n The easy part is remembering to call `super.init()` in each subclass's\n `init()` method. The harder part is fields. There is nothing preventing two\n classes in the inheritance chain from accidentally claiming the same field\n name. When this happens, they will step on each other's fields and possibly\n leave you with an instance in a broken state.\n\n If Lox was your language, how would you address this, if at all? If you\n would change the language, implement your change.\n\n2. Our copy-down inheritance optimization is valid only because Lox does not\n permit you to modify a class's methods after its declaration. This means we\n don't have to worry about the copied methods in the subclass getting out of\n sync with later changes to the superclass.\n\n Other languages, like Ruby, *do* allow classes to be modified after the\n fact. How do implementations of languages like that support class\n modification while keeping method resolution efficient?\n\n3. In the [jlox chapter on inheritance][inheritance], we had a challenge to\n implement the BETA language's approach to method overriding. Solve the\n challenge again, but this time in clox. Here's the description of the\n previous challenge:\n\n In Lox, as in most other object-oriented languages, when looking up a\n method, we start at the bottom of the class hierarchy and work our way up --\n a subclass's method is preferred over a superclass's. In order to get to the\n superclass method from within an overriding method, you use `super`.\n\n The language [BETA][] takes the [opposite approach][inner]. When you call a\n method, it starts at the *top* of the class hierarchy and works *down*. A\n superclass method wins over a subclass method. In order to get to the\n subclass method, the superclass method can call `inner`, which is sort of\n like the inverse of `super`. It chains to the next method down the\n hierarchy.\n\n The superclass method controls when and where the subclass is allowed to\n refine its behavior. If the superclass method doesn't call `inner` at all,\n then the subclass has no way of overriding or modifying the superclass's\n behavior.\n\n Take out Lox's current overriding and `super` behavior, and replace it with\n BETA's semantics. In short:\n\n * When calling a method on a class, the method *highest* on the\n class's inheritance chain takes precedence.\n\n * Inside the body of a method, a call to `inner` looks for a method with\n the same name in the nearest subclass along the inheritance chain\n between the class containing the `inner` and the class of `this`. If\n there is no matching method, the `inner` call does nothing.\n\n For example:\n\n ```lox\n class Doughnut {\n cook() {\n print \"Fry until golden brown.\";\n inner();\n print \"Place in a nice box.\";\n }\n }\n\n class BostonCream < Doughnut {\n cook() {\n print \"Pipe full of custard and coat with chocolate.\";\n }\n }\n\n BostonCream().cook();\n ```\n\n This should print:\n\n ```text\n Fry until golden brown.\n Pipe full of custard and coat with chocolate.\n Place in a nice box.\n ```\n\n Since clox is about not just implementing Lox, but doing so with good\n performance, this time around try to solve the challenge with an eye towards\n efficiency.\n\n[inheritance]: inheritance.html\n[inner]: http://journal.stuffwithstuff.com/2012/12/19/the-impoliteness-of-overriding-methods/\n[beta]: https://beta.cs.au.dk/\n\n
\n" }, { "path": "book/the-lox-language.md", "content": "> What nicer thing can you do for somebody than make them breakfast?\n>\n> Anthony Bourdain\n\nWe'll spend the rest of this book illuminating every dark and sundry corner of\nthe Lox language, but it seems cruel to have you immediately start grinding out\ncode for the interpreter without at least a glimpse of what we're going to end\nup with.\n\nAt the same time, I don't want to drag you through reams of language lawyering\nand specification-ese before you get to touch your text editor. So this will be a gentle, friendly introduction to\nLox. It will leave out a lot of details and edge cases. We've got plenty of time\nfor those later.\n\n\n\n## Hello, Lox\n\nHere's your very first taste of Lox:\n\n\n\n```lox\n// Your first Lox program!\nprint \"Hello, world!\";\n```\n\nAs that `//` line comment and the trailing semicolon imply, Lox's syntax is a\nmember of the C family. (There are no parentheses around the string because\n`print` is a built-in statement, and not a library function.)\n\nNow, I won't claim that C has a *great* syntax. If we\nwanted something elegant, we'd probably mimic Pascal or Smalltalk. If we wanted\nto go full Scandinavian-furniture-minimalism, we'd do a Scheme. Those all have\ntheir virtues.\n\n\n\nWhat C-like syntax has instead is something you'll often find more valuable\nin a language: *familiarity*. I know you are already comfortable with that style\nbecause the two languages we'll be using to *implement* Lox -- Java and C --\nalso inherit it. Using a similar syntax for Lox gives you one less thing to\nlearn.\n\n## A High-Level Language\n\nWhile this book ended up bigger than I was hoping, it's still not big enough to\nfit a huge language like Java in it. In order to fit two complete\nimplementations of Lox in these pages, Lox itself has to be pretty compact.\n\nWhen I think of languages that are small but useful, what comes to mind are\nhigh-level \"scripting\" languages like JavaScript, Scheme,\nand Lua. Of those three, Lox looks most like JavaScript, mainly because most\nC-syntax languages do. As we'll learn later, Lox's approach to scoping hews\nclosely to Scheme. The C flavor of Lox we'll build in [Part III][] is heavily\nindebted to Lua's clean, efficient implementation.\n\n[part iii]: a-bytecode-virtual-machine.html\n\n\n\nLox shares two other aspects with those three languages:\n\n### Dynamic typing\n\nLox is dynamically typed. Variables can store values of any type, and a single\nvariable can even store values of different types at different times. If you try\nto perform an operation on values of the wrong type -- say, dividing a number by\na string -- then the error is detected and reported at runtime.\n\nThere are plenty of reasons to like static types, but\nthey don't outweigh the pragmatic reasons to pick dynamic types for Lox. A\nstatic type system is a ton of work to learn and implement. Skipping it gives\nyou a simpler language and a shorter book. We'll get our interpreter up and\nexecuting bits of code sooner if we defer our type checking to runtime.\n\n\n\n### Automatic memory management\n\nHigh-level languages exist to eliminate error-prone, low-level drudgery, and what\ncould be more tedious than manually managing the allocation and freeing of\nstorage? No one rises and greets the morning sun with, \"I can't wait to figure\nout the correct place to call `free()` for every byte of memory I allocate\ntoday!\"\n\nThere are two main techniques for managing memory:\n**reference counting** and **tracing garbage collection** (usually just called\n**garbage collection** or **GC**). Ref counters are much simpler to implement --\nI think that's why Perl, PHP, and Python all started out using them. But, over\ntime, the limitations of ref counting become too troublesome. All of those\nlanguages eventually ended up adding a full tracing GC, or at least enough of\none to clean up object cycles.\n\n\n\nTracing garbage collection has a fearsome reputation. It *is* a little harrowing\nworking at the level of raw memory. Debugging a GC can sometimes leave you\nseeing hex dumps in your dreams. But, remember, this book is about dispelling\nmagic and slaying those monsters, so we *are* going to write our own garbage\ncollector. I think you'll find the algorithm is quite simple and a lot of fun to\nimplement.\n\n## Data Types\n\nIn Lox's little universe, the atoms that make up all matter are the built-in\ndata types. There are only a few:\n\n* **Booleans.** You can't code without logic and you\n can't logic without Boolean values. \"True\" and \"false\", the yin and yang of\n software. Unlike some ancient languages that repurpose an existing type to\n represent truth and falsehood, Lox has a dedicated Boolean type. We may\n be roughing it on this expedition, but we aren't *savages*.\n\n \n\n There are two Boolean values, obviously, and a literal for each one.\n\n ```lox\n true; // Not false.\n false; // Not *not* false.\n ```\n\n* **Numbers.** Lox has only one kind of number: double-precision floating\n point. Since floating-point numbers can also represent a wide range of\n integers, that covers a lot of territory, while keeping things simple.\n\n Full-featured languages have lots of syntax for numbers -- hexadecimal,\n scientific notation, octal, all sorts of fun stuff. We'll settle for basic\n integer and decimal literals.\n\n ```lox\n 1234; // An integer.\n 12.34; // A decimal number.\n ```\n\n* **Strings.** We've already seen one string literal in the first example.\n Like most languages, they are enclosed in double quotes.\n\n ```lox\n \"I am a string\";\n \"\"; // The empty string.\n \"123\"; // This is a string, not a number.\n ```\n\n As we'll see when we get to implementing them, there is quite a lot of\n complexity hiding in that innocuous sequence of characters.\n\n \n\n* **Nil.** There's one last built-in value who's never invited to the party\n but always seems to show up. It represents \"no value\". It's called \"null\" in\n many other languages. In Lox we spell it `nil`. (When we get to implementing\n it, that will help distinguish when we're talking about Lox's `nil` versus\n Java or C's `null`.)\n\n There are good arguments for not having a null value in a language since\n null pointer errors are the scourge of our industry. If we were doing a\n statically typed language, it would be worth trying to ban it. In a\n dynamically typed one, though, eliminating it is often more annoying\n than having it.\n\n## Expressions\n\nIf built-in data types and their literals are atoms, then **expressions** must\nbe the molecules. Most of these will be familiar.\n\n### Arithmetic\n\nLox features the basic arithmetic operators you know and love from C and other\nlanguages:\n\n```lox\nadd + me;\nsubtract - me;\nmultiply * me;\ndivide / me;\n```\n\nThe subexpressions on either side of the operator are **operands**. Because\nthere are *two* of them, these are called **binary** operators. (It has nothing\nto do with the ones-and-zeroes use of \"binary\".) Because the operator is fixed *in* the middle of the operands, these are also\ncalled **infix** operators (as opposed to **prefix** operators where the\noperator comes before the operands, and **postfix** where it comes after).\n\n\n\nOne arithmetic operator is actually *both* an infix and a prefix one. The `-`\noperator can also be used to negate a number.\n\n```lox\n-negateMe;\n```\n\nAll of these operators work on numbers, and it's an error to pass any other\ntypes to them. The exception is the `+` operator -- you can also pass it two\nstrings to concatenate them.\n\n### Comparison and equality\n\nMoving along, we have a few more operators that always return a Boolean result.\nWe can compare numbers (and only numbers), using Ye Olde Comparison Operators.\n\n```lox\nless < than;\nlessThan <= orEqual;\ngreater > than;\ngreaterThan >= orEqual;\n```\n\nWe can test two values of any kind for equality or inequality.\n\n```lox\n1 == 2; // false.\n\"cat\" != \"dog\"; // true.\n```\n\nEven different types.\n\n```lox\n314 == \"pi\"; // false.\n```\n\nValues of different types are *never* equivalent.\n\n```lox\n123 == \"123\"; // false.\n```\n\nI'm generally against implicit conversions.\n\n### Logical operators\n\nThe not operator, a prefix `!`, returns `false` if its operand is true, and vice\nversa.\n\n```lox\n!true; // false.\n!false; // true.\n```\n\nThe other two logical operators really are control flow constructs in the guise\nof expressions. An `and` expression determines if two\nvalues are *both* true. It returns the left operand if it's false, or the\nright operand otherwise.\n\n```lox\ntrue and false; // false.\ntrue and true; // true.\n```\n\nAnd an `or` expression determines if *either* of two values (or both) are true.\nIt returns the left operand if it is true and the right operand otherwise.\n\n```lox\nfalse or false; // false.\ntrue or false; // true.\n```\n\n\n\nThe reason `and` and `or` are like control flow structures is that they\n**short-circuit**. Not only does `and` return the left operand if it is false,\nit doesn't even *evaluate* the right one in that case. Conversely\n(contrapositively?), if the left operand of an `or` is true, the right is\nskipped.\n\n### Precedence and grouping\n\nAll of these operators have the same precedence and associativity that you'd\nexpect coming from C. (When we get to parsing, we'll get *way* more precise\nabout that.) In cases where the precedence isn't what you want, you can use `()`\nto group stuff.\n\n```lox\nvar average = (min + max) / 2;\n```\n\nSince they aren't very technically interesting, I've cut the remainder of the\ntypical operator menagerie out of our little language. No bitwise, shift,\nmodulo, or conditional operators. I'm not grading you, but you will get bonus\npoints in my heart if you augment your own implementation of Lox with them.\n\nThose are the expression forms (except for a couple related to specific features\nthat we'll get to later), so let's move up a level.\n\n## Statements\n\nNow we're at statements. Where an expression's main job is to produce a *value*,\na statement's job is to produce an *effect*. Since, by definition, statements\ndon't evaluate to a value, to be useful they have to otherwise change the world\nin some way -- usually modifying some state, reading input, or producing output.\n\nYou've seen a couple of kinds of statements already. The first one was:\n\n```lox\nprint \"Hello, world!\";\n```\n\nA `print` statement evaluates a single expression\nand displays the result to the user. You've also seen some statements like:\n\n\n\n```lox\n\"some expression\";\n```\n\nAn expression followed by a semicolon (`;`) promotes the expression to\nstatement-hood. This is called (imaginatively enough), an **expression\nstatement**.\n\nIf you want to pack a series of statements where a single one is expected, you\ncan wrap them up in a **block**.\n\n```lox\n{\n print \"One statement.\";\n print \"Two statements.\";\n}\n```\n\nBlocks also affect scoping, which leads us to the next section...\n\n## Variables\n\nYou declare variables using `var` statements. If you omit the initializer, the variable's value defaults to `nil`.\n\n\n\n```lox\nvar imAVariable = \"here is my value\";\nvar iAmNil;\n```\n\nOnce declared, you can, naturally, access and assign a variable using its name.\n\n\n\n```lox\nvar breakfast = \"bagels\";\nprint breakfast; // \"bagels\".\nbreakfast = \"beignets\";\nprint breakfast; // \"beignets\".\n```\n\n\n\nI won't get into the rules for variable scope here, because we're going to spend\na surprising amount of time in later chapters mapping every square inch of the\nrules. In most cases, it works like you would expect coming from C or Java.\n\n## Control Flow\n\nIt's hard to write useful programs if you can't skip\nsome code or execute some more than once. That means control flow. In addition\nto the logical operators we already covered, Lox lifts three statements straight\nfrom C.\n\n\n\nAn `if` statement executes one of two statements based on some condition.\n\n```lox\nif (condition) {\n print \"yes\";\n} else {\n print \"no\";\n}\n```\n\nA `while` loop executes the body repeatedly as long as\nthe condition expression evaluates to true.\n\n```lox\nvar a = 1;\nwhile (a < 10) {\n print a;\n a = a + 1;\n}\n```\n\n\n\nFinally, we have `for` loops.\n\n```lox\nfor (var a = 1; a < 10; a = a + 1) {\n print a;\n}\n```\n\nThis loop does the same thing as the previous `while` loop. Most modern\nlanguages also have some sort of `for-in` or\n`foreach` loop for explicitly iterating over various sequence types. In a real\nlanguage, that's nicer than the crude C-style `for` loop we got here. Lox keeps\nit basic.\n\n\n\n## Functions\n\nA function call expression looks the same as it does in C.\n\n```lox\nmakeBreakfast(bacon, eggs, toast);\n```\n\nYou can also call a function without passing anything to it.\n\n```lox\nmakeBreakfast();\n```\n\nUnlike in, say, Ruby, the parentheses are mandatory in this case. If you leave them\noff, the name doesn't *call* the function, it just refers to it.\n\nA language isn't very fun if you can't define your own functions. In Lox, you do\nthat with `fun`.\n\n\n\n```lox\nfun printSum(a, b) {\n print a + b;\n}\n```\n\nNow's a good time to clarify some terminology. Some\npeople throw around \"parameter\" and \"argument\" like they are interchangeable\nand, to many, they are. We're going to spend a lot of time splitting the finest\nof downy hairs around semantics, so let's sharpen our words. From here on out:\n\n* An **argument** is an actual value you pass to a function when you call it.\n So a function *call* has an *argument* list. Sometimes you hear **actual\n parameter** used for these.\n\n* A **parameter** is a variable that holds the value of the argument inside\n the body of the function. Thus, a function *declaration* has a *parameter*\n list. Others call these **formal parameters** or simply **formals**.\n\n\n\nThe body of a function is always a block. Inside it, you can return a value\nusing a `return` statement.\n\n```lox\nfun returnSum(a, b) {\n return a + b;\n}\n```\n\nIf execution reaches the end of the block without hitting a `return`, it\nimplicitly returns `nil`.\n\n\n\n### Closures\n\nFunctions are *first class* in Lox, which just means they are real values that\nyou can get a reference to, store in variables, pass around, etc. This works:\n\n```lox\nfun addPair(a, b) {\n return a + b;\n}\n\nfun identity(a) {\n return a;\n}\n\nprint identity(addPair)(1, 2); // Prints \"3\".\n```\n\nSince function declarations are statements, you can declare local functions\ninside another function.\n\n```lox\nfun outerFunction() {\n fun localFunction() {\n print \"I'm local!\";\n }\n\n localFunction();\n}\n```\n\nIf you combine local functions, first-class functions, and block scope, you run\ninto this interesting situation:\n\n```lox\nfun returnFunction() {\n var outside = \"outside\";\n\n fun inner() {\n print outside;\n }\n\n return inner;\n}\n\nvar fn = returnFunction();\nfn();\n```\n\nHere, `inner()` accesses a local variable declared outside of its body in the\nsurrounding function. Is this kosher? Now that lots of languages have borrowed\nthis feature from Lisp, you probably know the answer is yes.\n\nFor that to work, `inner()` has to \"hold on\" to references to any surrounding\nvariables that it uses so that they stay around even after the outer function\nhas returned. We call functions that do this **closures**. These days, the term is often used for *any*\nfirst-class function, though it's sort of a misnomer if the function doesn't\nhappen to close over any variables.\n\n\n\nAs you can imagine, implementing these adds some complexity because we can no\nlonger assume variable scope works strictly like a stack where local variables\nevaporate the moment the function returns. We're going to have a fun time\nlearning how to make these work correctly and efficiently.\n\n## Classes\n\nSince Lox has dynamic typing, lexical (roughly, \"block\") scope, and closures,\nit's about halfway to being a functional language. But as you'll see, it's\n*also* about halfway to being an object-oriented language. Both paradigms have a\nlot going for them, so I thought it was worth covering some of each.\n\nSince classes have come under fire for not living up to their hype, let me first\nexplain why I put them into Lox and this book. There are really two questions:\n\n### Why might any language want to be object oriented?\n\nNow that object-oriented languages like Java have sold out and only play arena\nshows, it's not cool to like them anymore. Why would anyone make a *new*\nlanguage with objects? Isn't that like releasing music on 8-track?\n\nIt is true that the \"all inheritance all the time\" binge of the '90s produced\nsome monstrous class hierarchies, but **object-oriented programming** (**OOP**)\nis still pretty rad. Billions of lines of successful code have been written in\nOOP languages, shipping millions of apps to happy users. Likely a majority of\nworking programmers today are using an object-oriented language. They can't all\nbe *that* wrong.\n\nIn particular, for a dynamically typed language, objects are pretty handy. We\nneed *some* way of defining compound data types to bundle blobs of stuff\ntogether.\n\nIf we can also hang methods off of those, then we avoid the need to prefix all\nof our functions with the name of the data type they operate on to avoid\ncolliding with similar functions for different types. In, say, Racket, you end\nup having to name your functions like `hash-copy` (to copy a hash table) and\n`vector-copy` (to copy a vector) so that they don't step on each other. Methods\nare scoped to the object, so that problem goes away.\n\n### Why is Lox object oriented?\n\nI could claim objects are groovy but still out of scope for the book. Most\nprogramming language books, especially ones that try to implement a whole\nlanguage, leave objects out. To me, that means the topic isn't well covered.\nWith such a widespread paradigm, that omission makes me sad.\n\nGiven how many of us spend all day *using* OOP languages, it seems like the\nworld could use a little documentation on how to *make* one. As you'll see, it\nturns out to be pretty interesting. Not as hard as you might fear, but not as\nsimple as you might presume, either.\n\n### Classes or prototypes\n\nWhen it comes to objects, there are actually two approaches to them, [classes][]\nand [prototypes][]. Classes came first, and are more common thanks to C++, Java,\nC#, and friends. Prototypes were a virtually forgotten offshoot until JavaScript\naccidentally took over the world.\n\n[classes]: https://en.wikipedia.org/wiki/Class-based_programming\n[prototypes]: https://en.wikipedia.org/wiki/Prototype-based_programming\n\nIn class-based languages, there are two core concepts: instances and classes.\nInstances store the state for each object and have a reference to the instance's\nclass. Classes contain the methods and inheritance chain. To call a method on an\ninstance, there is always a level of indirection. You look up the instance's class and then you find the method\n*there*:\n\n\n\n\"How\n\nPrototype-based languages merge these two concepts.\nThere are only objects -- no classes -- and each individual object may contain\nstate and methods. Objects can directly inherit from each other (or \"delegate\nto\" in prototypal lingo):\n\n\n\n\"How\n\nThis means that in some ways prototypal languages are more fundamental than\nclasses. They are really neat to implement because they're *so* simple. Also,\nthey can express lots of unusual patterns that classes steer you away from.\n\nBut I've looked at a *lot* of code written in prototypal languages -- including\n[some of my own devising][finch]. Do you know what people generally do with all\nof the power and flexibility of prototypes? ...They use them to reinvent\nclasses.\n\n[finch]: http://finch.stuffwithstuff.com/\n\nI don't know *why* that is, but people naturally seem to prefer a class-based\n(Classic? Classy?) style. Prototypes *are* simpler in the language, but they\nseem to accomplish that only by pushing the\ncomplexity onto the user. So, for Lox, we'll save our users the trouble and bake\nclasses right in.\n\n\n\n### Classes in Lox\n\nEnough rationale, let's see what we actually have. Classes encompass a\nconstellation of features in most languages. For Lox, I've selected what I think\nare the brightest stars. You declare a class and its methods like so:\n\n```lox\nclass Breakfast {\n cook() {\n print \"Eggs a-fryin'!\";\n }\n\n serve(who) {\n print \"Enjoy your breakfast, \" + who + \".\";\n }\n}\n```\n\nThe body of a class contains its methods. They look like function declarations\nbut without the `fun` keyword. When the class\ndeclaration is executed, Lox creates a class object and stores that in a\nvariable named after the class. Just like functions, classes are first class in\nLox.\n\n\n\n```lox\n// Store it in variables.\nvar someVariable = Breakfast;\n\n// Pass it to functions.\nsomeFunction(Breakfast);\n```\n\nNext, we need a way to create instances. We could add some sort of `new`\nkeyword, but to keep things simple, in Lox the class itself is a factory\nfunction for instances. Call a class like a function, and it produces a new\ninstance of itself.\n\n```lox\nvar breakfast = Breakfast();\nprint breakfast; // \"Breakfast instance\".\n```\n\n### Instantiation and initialization\n\nClasses that only have behavior aren't super useful. The idea behind\nobject-oriented programming is encapsulating behavior *and state* together. To\ndo that, you need fields. Lox, like other dynamically typed languages, lets you\nfreely add properties onto objects.\n\n```lox\nbreakfast.meat = \"sausage\";\nbreakfast.bread = \"sourdough\";\n```\n\nAssigning to a field creates it if it doesn't already exist.\n\nIf you want to access a field or method on the current object from within a\nmethod, you use good old `this`.\n\n```lox\nclass Breakfast {\n serve(who) {\n print \"Enjoy your \" + this.meat + \" and \" +\n this.bread + \", \" + who + \".\";\n }\n\n // ...\n}\n```\n\nPart of encapsulating data within an object is ensuring the object is in a valid\nstate when it's created. To do that, you can define an initializer. If your\nclass has a method named `init()`, it is called automatically when the object is\nconstructed. Any parameters passed to the class are forwarded to its\ninitializer.\n\n```lox\nclass Breakfast {\n init(meat, bread) {\n this.meat = meat;\n this.bread = bread;\n }\n\n // ...\n}\n\nvar baconAndToast = Breakfast(\"bacon\", \"toast\");\nbaconAndToast.serve(\"Dear Reader\");\n// \"Enjoy your bacon and toast, Dear Reader.\"\n```\n\n### Inheritance\n\nEvery object-oriented language lets you not only define methods, but reuse them\nacross multiple classes or objects. For that, Lox supports single inheritance.\nWhen you declare a class, you can specify a class that it inherits from using a less-than\n(`<`) operator.\n\n```lox\nclass Brunch < Breakfast {\n drink() {\n print \"How about a Bloody Mary?\";\n }\n}\n```\n\n\n\nHere, Brunch is the **derived class** or **subclass**, and Breakfast is the\n**base class** or **superclass**.\n\nEvery method defined in the superclass is also available to its subclasses.\n\n```lox\nvar benedict = Brunch(\"ham\", \"English muffin\");\nbenedict.serve(\"Noble Reader\");\n```\n\nEven the `init()` method gets inherited. In practice,\nthe subclass usually wants to define its own `init()` method too. But the\noriginal one also needs to be called so that the superclass can maintain its\nstate. We need some way to call a method on our own *instance* without hitting\nour own *methods*.\n\n\n\nAs in Java, you use `super` for that.\n\n```lox\nclass Brunch < Breakfast {\n init(meat, bread, drink) {\n super.init(meat, bread);\n this.drink = drink;\n }\n}\n```\n\nThat's about it for object orientation. I tried to keep the feature set minimal.\nThe structure of the book did force one compromise. Lox is not a *pure*\nobject-oriented language. In a true OOP language every object is an instance of\na class, even primitive values like numbers and Booleans.\n\nBecause we don't implement classes until well after we start working with the\nbuilt-in types, that would have been hard. So values of primitive types aren't\nreal objects in the sense of being instances of classes. They don't have methods\nor properties. If I were trying to make Lox a real language for real users, I\nwould fix that.\n\n## The Standard Library\n\nWe're almost done. That's the whole language, so all that's left is the \"core\"\nor \"standard\" library -- the set of functionality that is implemented directly\nin the interpreter and that all user-defined behavior is built on top of.\n\nThis is the saddest part of Lox. Its standard library goes beyond minimalism and\nveers close to outright nihilism. For the sample code in the book, we only need\nto demonstrate that code is running and doing what it's supposed to do. For\nthat, we already have the built-in `print` statement.\n\nLater, when we start optimizing, we'll write some benchmarks and see how long it\ntakes to execute code. That means we need to track time, so we'll define one\nbuilt-in function, `clock()`, that returns the number of seconds since the\nprogram started.\n\nAnd... that's it. I know, right? It's embarrassing.\n\nIf you wanted to turn Lox into an actual useful language, the very first thing\nyou should do is flesh this out. String manipulation, trigonometric functions,\nfile I/O, networking, heck, even *reading input from the user* would help. But we\ndon't need any of that for this book, and adding it wouldn't teach you anything\ninteresting, so I've left it out.\n\nDon't worry, we'll have plenty of exciting stuff in the language itself to keep\nus busy.\n\n
\n\n## Challenges\n\n1. Write some sample Lox programs and run them (you can use the implementations\n of Lox in [my repository][repo]). Try to come up with edge case behavior I\n didn't specify here. Does it do what you expect? Why or why not?\n\n2. This informal introduction leaves a *lot* unspecified. List several open\n questions you have about the language's syntax and semantics. What do you\n think the answers should be?\n\n3. Lox is a pretty tiny language. What features do you think it is missing that\n would make it annoying to use for real programs? (Aside from the standard\n library, of course.)\n\n
\n\n
\n\n## Design Note: Expressions and Statements\n\nLox has both expressions and statements. Some languages omit the latter.\nInstead, they treat declarations and control flow constructs as expressions too.\nThese \"everything is an expression\" languages tend to have functional pedigrees\nand include most Lisps, SML, Haskell, Ruby, and CoffeeScript.\n\nTo do that, for each \"statement-like\" construct in the language, you need to\ndecide what value it evaluates to. Some of those are easy:\n\n* An `if` expression evaluates to the result of whichever branch is chosen.\n Likewise, a `switch` or other multi-way branch evaluates to whichever case\n is picked.\n\n* A variable declaration evaluates to the value of the variable.\n\n* A block evaluates to the result of the last expression in the sequence.\n\nSome get a little stranger. What should a loop evaluate to? A `while` loop in\nCoffeeScript evaluates to an array containing each element that the body\nevaluated to. That can be handy, or a waste of memory if you don't need the\narray.\n\nYou also have to decide how these statement-like expressions compose with other\nexpressions -- you have to fit them into the grammar's precedence table. For\nexample, Ruby allows:\n\n```ruby\nputs 1 + if true then 2 else 3 end + 4\n```\n\nIs this what you'd expect? Is it what your *users* expect? How does this affect\nhow you design the syntax for your \"statements\"? Note that Ruby has an explicit\n`end` to tell when the `if` expression is complete. Without it, the `+ 4` would\nlikely be parsed as part of the `else` clause.\n\nTurning every statement into an expression forces you to answer a few hairy\nquestions like that. In return, you eliminate some redundancy. C has both blocks\nfor sequencing statements, and the comma operator for sequencing expressions. It\nhas both the `if` statement and the `?:` conditional operator. If everything was\nan expression in C, you could unify each of those.\n\nLanguages that do away with statements usually also feature **implicit returns**\n-- a function automatically returns whatever value its body evaluates to without\nneed for some explicit `return` syntax. For small functions and methods, this is\nreally handy. In fact, many languages that do have statements have added syntax\nlike `=>` to be able to define functions whose body is the result of evaluating\na single expression.\n\nBut making *all* functions work that way can be a little strange. If you aren't\ncareful, your function will leak a return value even if you only intend it to\nproduce a side effect. In practice, though, users of these languages don't find\nit to be a problem.\n\nFor Lox, I gave it statements for prosaic reasons. I picked a C-like syntax for\nfamiliarity's sake, and trying to take the existing C statement syntax and\ninterpret it like expressions gets weird pretty fast.\n\n
\n" }, { "path": "book/types-of-values.md", "content": "> When you are a Bear of Very Little Brain, and you Think of Things, you find\n> sometimes that a Thing which seemed very Thingish inside you is quite\n> different when it gets out into the open and has other people looking at it.\n>\n> A. A. Milne, Winnie-the-Pooh\n\nThe past few chapters were huge, packed full of complex techniques and pages of\ncode. In this chapter, there's only one new concept to learn and a scattering of\nstraightforward code. You've earned a respite.\n\nLox is dynamically typed. A single variable can\nhold a Boolean, number, or string at different points in time. At least, that's\nthe idea. Right now, in clox, all values are numbers. By the end of the chapter,\nit will also support Booleans and `nil`. While those aren't super interesting,\nthey force us to figure out how our value representation can dynamically handle\ndifferent types.\n\n\n\n## Tagged Unions\n\nThe nice thing about working in C is that we can build our data structures from\nthe raw bits up. The bad thing is that we *have* to do that. C doesn't give you\nmuch for free at compile time and even less at runtime. As far as C is\nconcerned, the universe is an undifferentiated array of bytes. It's up to us to\ndecide how many of those bytes to use and what they mean.\n\nIn order to choose a value representation, we need to answer two key questions:\n\n1. **How do we represent the type of a value?** If you try to, say, multiply a\n number by `true`, we need to detect that error at runtime and report it. In\n order to do that, we need to be able to tell what a value's type is.\n\n2. **How do we store the value itself?** We need to not only be able to tell\n that three is a number, but that it's different from the number four. I\n know, seems obvious, right? But we're operating at a level where it's good\n to spell these things out.\n\nSince we're not just designing this language but building it ourselves, when\nanswering these two questions we also have to keep in mind the implementer's\neternal quest: to do it *efficiently*.\n\nLanguage hackers over the years have come up with a variety of clever ways to\npack the above information into as few bits as possible. For now, we'll start\nwith the simplest, classic solution: a **tagged union**. A value contains two\nparts: a type \"tag\", and a payload for the actual value. To store the value's\ntype, we define an enum for each kind of value the VM supports.\n\n^code value-type (2 before, 1 after)\n\n\n\nFor now, we have only a couple of cases, but this will grow as we add strings,\nfunctions, and classes to clox. In addition to the type, we also need to store\nthe data for the value -- the `double` for a number, `true` or `false` for a\nBoolean. We could define a struct with fields for each possible type.\n\n\"A\n\nBut this is a waste of memory. A value can't simultaneously be both a number and\na Boolean. So at any point in time, only one of those fields will be used. C\nlets you optimize this by defining a union. A union\nlooks like a struct except that all of its fields overlap in memory.\n\n\n\n\"A\n\nThe size of a union is the size of its largest field. Since the fields all reuse\nthe same bits, you have to be very careful when working with them. If you store\ndata using one field and then access it using another, you will reinterpret what the underlying bits\nmean.\n\n\n\nAs the name \"tagged union\" implies, our new value representation combines these\ntwo parts into a single struct.\n\n^code value (2 before, 2 after)\n\nThere's a field for the type tag, and then a second field containing the union\nof all of the underlying values. On a 64-bit machine with a typical C compiler,\nthe layout looks like this:\n\n\n\n\"The\n\nThe four-byte type tag comes first, then the union. Most architectures prefer\nvalues be aligned to their size. Since the union field contains an eight-byte\ndouble, the compiler adds four bytes of padding after\nthe type field to keep that double on the nearest eight-byte boundary. That\nmeans we're effectively spending eight bytes on the type tag, which only needs\nto represent a number between zero and three. We could stuff the enum in a\nsmaller size, but all that would do is increase the padding.\n\n\n\nSo our Values are 16 bytes, which seems a little large. We'll improve it\n[later][optimization]. In the meantime, they're still small enough to store on\nthe C stack and pass around by value. Lox's semantics allow that because the\nonly types we support so far are **immutable**. If we pass a copy of a Value\ncontaining the number three to some function, we don't need to worry about the\ncaller seeing modifications to the value. You can't \"modify\" three. It's three\nforever.\n\n[optimization]: optimization.html\n\n## Lox Values and C Values\n\nThat's our new value representation, but we aren't done. Right now, the rest of\nclox assumes Value is an alias for `double`. We have code that does a straight C\ncast from one to the other. That code is all broken now. So sad.\n\nWith our new representation, a Value can *contain* a double, but it's not\n*equivalent* to it. There is a mandatory conversion step to get from one to the\nother. We need to go through the code and insert those conversions to get clox\nworking again.\n\nWe'll implement these conversions as a handful of macros, one for each type and\noperation. First, to promote a native C value to a clox Value:\n\n^code value-macros (1 before, 2 after)\n\nEach one of these takes a C value of the appropriate type and produces a Value\nthat has the correct type tag and contains the underlying value. This hoists\nstatically typed values up into clox's dynamically typed universe. In order to\n*do* anything with a Value, though, we need to unpack it and get the C value\nback out.\n\n^code as-macros (1 before, 2 after)\n\n\n\nThese macros go in the opposite direction. Given a\nValue of the right type, they unwrap it and return the corresponding raw C\nvalue. The \"right type\" part is important! These macros directly access the\nunion fields. If we were to do something like:\n\n```c\nValue value = BOOL_VAL(true);\ndouble number = AS_NUMBER(value);\n```\n\nThen we may open a smoldering portal to the Shadow Realm. It's not safe to use\nany of the `AS_` macros unless we know the Value contains the appropriate type.\nTo that end, we define a last few macros to check a Value's type.\n\n^code is-macros (1 before, 2 after)\n\nThese macros return `true` if the Value has that\ntype. Any time we call one of the `AS_` macros, we need to guard it behind a\ncall to one of these first. With these eight macros, we can now safely shuttle\ndata between Lox's dynamic world and C's static one.\n\n\n\n## Dynamically Typed Numbers\n\nWe've got our value representation and the tools to convert to and from it. All\nthat's left to get clox running again is to grind through the code and fix every\nplace where data moves across that boundary. This is one of those sections of\nthe book that isn't exactly mind-blowing, but I promised I'd show you every\nsingle line of code, so here we are.\n\nThe first values we create are the constants generated when we compile number\nliterals. After we convert the lexeme to a C double, we simply wrap it in a\nValue before storing it in the constant table.\n\n^code const-number-val (1 before, 1 after)\n\nOver in the runtime, we have a function to print values.\n\n^code print-number-value (1 before, 1 after)\n\nRight before we send the Value to `printf()`, we unwrap it and extract the\ndouble value. We'll revisit this function shortly to add the other types, but\nlet's get our existing code working first.\n\n### Unary negation and runtime errors\n\nThe next simplest operation is unary negation. It pops a value off the stack,\nnegates it, and pushes the result. Now that we have other types of values, we\ncan't assume the operand is a number anymore. The user could just as well do:\n\n```lox\nprint -false; // Uh...\n```\n\nWe need to handle that gracefully, which means it's time for *runtime errors*.\nBefore performing an operation that requires a certain type, we need to make\nsure the Value *is* that type.\n\nFor unary negation, the check looks like this:\n\n^code op-negate (1 before, 1 after)\n\nFirst, we check to see if the Value on top of the stack is a number. If it's\nnot, we report the runtime error and stop the\ninterpreter. Otherwise, we keep going. Only after this validation do we unwrap\nthe operand, negate it, wrap the result and push it.\n\n\n\nTo access the Value, we use a new little function.\n\n^code peek\n\nIt returns a Value from the stack but doesn't pop it.\nThe `distance` argument is how far down from the top of the stack to look: zero\nis the top, one is one slot down, etc.\n\n\n\nWe report the runtime error using a new function that we'll get a lot of mileage\nout of over the remainder of the book.\n\n^code runtime-error\n\nYou've certainly *called* variadic functions -- ones that take a varying number\nof arguments -- in C before: `printf()` is one. But you may not have *defined*\nyour own. This book isn't a C tutorial, so I'll\nskim over it here, but basically the `...` and `va_list` stuff let us pass an\narbitrary number of arguments to `runtimeError()`. It forwards those on to\n`vfprintf()`, which is the flavor of `printf()` that takes an explicit\n`va_list`.\n\n\n\nCallers can pass a format string to `runtimeError()` followed by a number of\narguments, just like they can when calling `printf()` directly. `runtimeError()`\nthen formats and prints those arguments. We won't take advantage of that in this\nchapter, but later chapters will produce formatted runtime error messages that\ncontain other data.\n\nAfter we show the hopefully helpful error message, we tell the user which line of their code was being executed when the error\noccurred. Since we left the tokens behind in the compiler, we look up the line\nin the debug information compiled into the chunk. If our compiler did its job\nright, that corresponds to the line of source code that the bytecode was\ncompiled from.\n\nWe look into the chunk's debug line array using the current bytecode instruction\nindex *minus one*. That's because the interpreter advances past each instruction\nbefore executing it. So, at the point that we call `runtimeError()`, the failed\ninstruction is the previous one.\n\n\n\nIn order to use `va_list` and the macros for working with it, we need to bring\nin a standard header.\n\n^code include-stdarg (1 after)\n\nWith this, our VM can not only do the right thing when we negate numbers (like\nit used to before we broke it), but it also gracefully handles erroneous\nattempts to negate other types (which we don't have yet, but still).\n\n### Binary arithmetic operators\n\nWe have our runtime error machinery in place now, so fixing the binary operators\nis easier even though they're more complex. We support four binary operators\ntoday: `+`, `-`, `*`, and `/`. The only difference between them is which\nunderlying C operator they use. To minimize redundant code between the four\noperators, we wrapped up the commonality in a big preprocessor macro that takes\nthe operator token as a parameter.\n\nThat macro seemed like overkill a [few chapters ago][], but we get the benefit\nfrom it today. It lets us add the necessary type checking and conversions in one\nplace.\n\n[few chapters ago]: a-virtual-machine.html#binary-operators\n\n^code binary-op (1 before, 2 after)\n\nYeah, I realize that's a monster of a macro. It's not what I'd normally consider\ngood C practice, but let's roll with it. The changes are similar to what we did\nfor unary negate. First, we check that the two operands are both numbers. If\neither isn't, we report a runtime error and yank the ejection seat lever.\n\nIf the operands are fine, we pop them both and unwrap them. Then we apply the\ngiven operator, wrap the result, and push it back on the stack. Note that we\ndon't wrap the result by directly using `NUMBER_VAL()`. Instead, the wrapper to\nuse is passed in as a macro parameter. For our\nexisting arithmetic operators, the result is a number, so we pass in the\n`NUMBER_VAL` macro.\n\n\n\n^code op-arithmetic (1 before, 1 after)\n\nSoon, I'll show you why we made the wrapping macro an argument.\n\n## Two New Types\n\nAll of our existing clox code is back in working order. Finally, it's time to\nadd some new types. We've got a running numeric calculator that now does a\nnumber of pointless paranoid runtime type checks. We can represent other types\ninternally, but there's no way for a user's program to ever create a Value of\none of those types.\n\nNot until now, that is. We'll start by adding compiler support for the three new\nliterals: `true`, `false`, and `nil`. They're all pretty simple, so we'll do all\nthree in a single batch.\n\nWith number literals, we had to deal with the fact that there are billions of\npossible numeric values. We attended to that by storing the literal's value in\nthe chunk's constant table and emitting a bytecode instruction that simply\nloaded that constant. We could do the same thing for the new types. We'd store,\nsay, `true`, in the constant table, and use an `OP_CONSTANT` to read it out.\n\nBut given that there are literally (heh) only three possible values we need to\nworry about with these new types, it's gratuitous -- and slow! -- to waste a two-byte instruction and a constant\ntable entry on them. Instead, we'll define three dedicated instructions to push\neach of these literals on the stack.\n\n\n\n^code literal-ops (1 before, 1 after)\n\nOur scanner already treats `true`, `false`, and `nil` as keywords, so we can\nskip right to the parser. With our table-based Pratt parser, we just need to\nslot parser functions into the rows associated with those keyword token types.\nWe'll use the same function in all three slots. Here:\n\n^code table-false (1 before, 1 after)\n\nHere:\n\n^code table-true (1 before, 1 after)\n\nAnd here:\n\n^code table-nil (1 before, 1 after)\n\nWhen the parser encounters `false`, `nil`, or `true`, in prefix position, it\ncalls this new parser function:\n\n^code parse-literal\n\nSince `parsePrecedence()` has already consumed the keyword token, all we need to\ndo is output the proper instruction. We figure that\nout based on the type of token we parsed. Our front end can now compile Boolean\nand nil literals to bytecode. Moving down the execution pipeline, we reach the\ninterpreter.\n\n\n\n^code interpret-literals (5 before, 1 after)\n\nThis is pretty self-explanatory. Each instruction summons the appropriate value\nand pushes it onto the stack. We shouldn't forget our disassembler either.\n\n^code disassemble-literals (2 before, 1 after)\n\nWith this in place, we can run this Earth-shattering program:\n\n```lox\ntrue\n```\n\nExcept that when the interpreter tries to print the result, it blows up. We need\nto extend `printValue()` to handle the new types too:\n\n^code print-value (1 before, 1 after)\n\nThere we go! Now we have some new types. They just aren't very useful yet. Aside\nfrom the literals, you can't really *do* anything with them. It will be a while\nbefore `nil` comes into play, but we can start putting Booleans to work in the\nlogical operators.\n\n### Logical not and falsiness\n\nThe simplest logical operator is our old exclamatory friend unary not.\n\n```lox\nprint !true; // \"false\"\n```\n\nThis new operation gets a new instruction.\n\n^code not-op (1 before, 1 after)\n\nWe can reuse the `unary()` parser function we wrote for unary negation to\ncompile a not expression. We just need to slot it into the parsing table.\n\n^code table-not (1 before, 1 after)\n\nBecause I knew we were going to do this, the `unary()` function already has a\nswitch on the token type to figure out which bytecode instruction to output. We\nmerely add another case.\n\n^code compile-not (1 before, 3 after)\n\nThat's it for the front end. Let's head over to the VM and conjure this\ninstruction into life.\n\n^code op-not (1 before, 1 after)\n\nLike our previous unary operator, it pops the one operand, performs the\noperation, and pushes the result. And, as we did there, we have to worry about\ndynamic typing. Taking the logical not of `true` is easy, but there's nothing\npreventing an unruly programmer from writing something like this:\n\n```lox\nprint !nil;\n```\n\nFor unary minus, we made it an error to negate anything that isn't a number. But Lox, like most scripting languages, is more\npermissive when it comes to `!` and other contexts where a Boolean is expected.\nThe rule for how other types are handled is called \"falsiness\", and we implement\nit here:\n\n\n\n^code is-falsey\n\nLox follows Ruby in that `nil` and `false` are falsey and every other value\nbehaves like `true`. We've got a new instruction we can generate, so we also\nneed to be able to *un*generate it in the disassembler.\n\n^code disassemble-not (2 before, 1 after)\n\n### Equality and comparison operators\n\nThat wasn't too bad. Let's keep the momentum going and knock out the equality\nand comparison operators too: `==`, `!=`, `<`, `>`, `<=`, and `>=`. That covers\nall of the operators that return Boolean results except the logical operators\n`and` and `or`. Since those need to short-circuit (basically do a little\ncontrol flow) we aren't ready for them yet.\n\nHere are the new instructions for those operators:\n\n^code comparison-ops (1 before, 1 after)\n\nWait, only three? What about `!=`, `<=`, and `>=`? We could create instructions\nfor those too. Honestly, the VM would execute faster if we did, so we *should*\ndo that if the goal is performance.\n\nBut my main goal is to teach you about bytecode compilers. I want you to start\ninternalizing the idea that the bytecode instructions don't need to closely\nfollow the user's source code. The VM has total freedom to use whatever\ninstruction set and code sequences it wants as long as they have the right\nuser-visible behavior.\n\nThe expression `a != b` has the same semantics as `!(a == b)`, so the compiler\nis free to compile the former as if it were the latter. Instead of a dedicated\n`OP_NOT_EQUAL` instruction, it can output an `OP_EQUAL` followed by an `OP_NOT`.\nLikewise, `a <= b` is the same as `!(a > b)` and `a >=\nb` is `!(a < b)`. Thus, we only need three new instructions.\n\n\n\nOver in the parser, though, we do have six new operators to slot into the parse\ntable. We use the same `binary()` parser function from before. Here's the row\nfor `!=`:\n\n^code table-equal (1 before, 1 after)\n\nThe remaining five operators are a little farther down in the table.\n\n^code table-comparisons (1 before, 1 after)\n\nInside `binary()` we already have a switch to generate the right bytecode for\neach token type. We add cases for the six new operators.\n\n^code comparison-operators (1 before, 1 after)\n\nThe `==`, `<`, and `>` operators output a single instruction. The others output\na pair of instructions, one to evalute the inverse operation, and then an\n`OP_NOT` to flip the result. Six operators for the price of three instructions!\n\nThat means over in the VM, our job is simpler. Equality is the most general\noperation.\n\n^code interpret-equal (1 before, 1 after)\n\nYou can evaluate `==` on any pair of objects, even objects of different types.\nThere's enough complexity that it makes sense to shunt that logic over to a\nseparate function. That function always returns a C `bool`, so we can safely\nwrap the result in a `BOOL_VAL`. The function relates to Values, so it lives\nover in the \"value\" module.\n\n^code values-equal-h (2 before, 1 after)\n\nAnd here's the implementation:\n\n^code values-equal\n\nFirst, we check the types. If the Values have different types, they are definitely not equal. Otherwise,\nwe unwrap the two Values and compare them directly.\n\n\n\nFor each value type, we have a separate case that handles comparing the value\nitself. Given how similar the cases are, you might wonder why we can't simply\n`memcmp()` the two Value structs and be done with it. The problem is that\nbecause of padding and different-sized union fields, a Value contains unused\nbits. C gives no guarantee about what is in those, so it's possible that two\nequal Values actually differ in memory that isn't used.\n\n\"The\n\n(You wouldn't believe how much pain I went through before learning this fact.)\n\nAnyway, as we add more types to clox, this function will grow new cases. For\nnow, these three are sufficient. The other comparison operators are easier since\nthey work only on numbers.\n\n^code interpret-comparison (3 before, 1 after)\n\nWe already extended the `BINARY_OP` macro to handle operators that return\nnon-numeric types. Now we get to use that. We pass in `BOOL_VAL` since the\nresult value type is Boolean. Otherwise, it's no different from plus or minus.\n\nAs always, the coda to today's aria is disassembling the new instructions.\n\n^code disassemble-comparison (2 before, 1 after)\n\nWith that, our numeric calculator has become something closer to a general\nexpression evaluator. Fire up clox and type in:\n\n```lox\n!(5 - 4 > 3 * 2 == !nil)\n```\n\nOK, I'll admit that's maybe not the most *useful* expression, but we're making\nprogress. We have one missing built-in type with its own literal form: strings.\nThose are much more complex because strings can vary in size. That tiny\ndifference turns out to have implications so large that we give strings [their\nvery own chapter][strings].\n\n[strings]: strings.html\n\n
\n\n## Challenges\n\n1. We could reduce our binary operators even further than we did here. Which\n other instructions can you eliminate, and how would the compiler cope with\n their absence?\n\n2. Conversely, we can improve the speed of our bytecode VM by adding more\n specific instructions that correspond to higher-level operations. What\n instructions would you define to speed up the kind of user code we added\n support for in this chapter?\n\n
\n" }, { "path": "book/welcome.md", "content": "This may be the beginning of a grand adventure. Programming languages encompass\na huge space to explore and play in. Plenty of room for your own creations to\nshare with others or just enjoy yourself. Brilliant computer scientists and\nsoftware engineers have spent entire careers traversing this land without ever\nreaching the end. If this book is your first entry into the country, welcome.\n\nThe pages of this book give you a guided tour through some of the world of\nlanguages. But before we strap on our hiking boots and venture out, we should\nfamiliarize ourselves with the territory. The chapters in this part introduce\nyou to the basic concepts used by programming languages and how those concepts\nare organized.\n\nWe will also get acquainted with Lox, the language we'll spend the rest of the\nbook implementing (twice).\n" }, { "path": "c/chunk.c", "content": "//> Chunks of Bytecode chunk-c\n#include \n\n#include \"chunk.h\"\n//> chunk-c-include-memory\n#include \"memory.h\"\n//< chunk-c-include-memory\n//> Garbage Collection chunk-include-vm\n#include \"vm.h\"\n//< Garbage Collection chunk-include-vm\n\nvoid initChunk(Chunk* chunk) {\n chunk->count = 0;\n chunk->capacity = 0;\n chunk->code = NULL;\n//> chunk-null-lines\n chunk->lines = NULL;\n//< chunk-null-lines\n//> chunk-init-constant-array\n initValueArray(&chunk->constants);\n//< chunk-init-constant-array\n}\n//> free-chunk\nvoid freeChunk(Chunk* chunk) {\n FREE_ARRAY(uint8_t, chunk->code, chunk->capacity);\n//> chunk-free-lines\n FREE_ARRAY(int, chunk->lines, chunk->capacity);\n//< chunk-free-lines\n//> chunk-free-constants\n freeValueArray(&chunk->constants);\n//< chunk-free-constants\n initChunk(chunk);\n}\n//< free-chunk\n/* Chunks of Bytecode write-chunk < Chunks of Bytecode write-chunk-with-line\nvoid writeChunk(Chunk* chunk, uint8_t byte) {\n*/\n//> write-chunk\n//> write-chunk-with-line\nvoid writeChunk(Chunk* chunk, uint8_t byte, int line) {\n//< write-chunk-with-line\n if (chunk->capacity < chunk->count + 1) {\n int oldCapacity = chunk->capacity;\n chunk->capacity = GROW_CAPACITY(oldCapacity);\n chunk->code = GROW_ARRAY(uint8_t, chunk->code,\n oldCapacity, chunk->capacity);\n//> write-chunk-line\n chunk->lines = GROW_ARRAY(int, chunk->lines,\n oldCapacity, chunk->capacity);\n//< write-chunk-line\n }\n\n chunk->code[chunk->count] = byte;\n//> chunk-write-line\n chunk->lines[chunk->count] = line;\n//< chunk-write-line\n chunk->count++;\n}\n//< write-chunk\n//> add-constant\nint addConstant(Chunk* chunk, Value value) {\n//> Garbage Collection add-constant-push\n push(value);\n//< Garbage Collection add-constant-push\n writeValueArray(&chunk->constants, value);\n//> Garbage Collection add-constant-pop\n pop();\n//< Garbage Collection add-constant-pop\n return chunk->constants.count - 1;\n}\n//< add-constant\n" }, { "path": "c/chunk.h", "content": "//> Chunks of Bytecode chunk-h\n#ifndef clox_chunk_h\n#define clox_chunk_h\n\n#include \"common.h\"\n//> chunk-h-include-value\n#include \"value.h\"\n//< chunk-h-include-value\n//> op-enum\n\ntypedef enum {\n//> op-constant\n OP_CONSTANT,\n//< op-constant\n//> Types of Values literal-ops\n OP_NIL,\n OP_TRUE,\n OP_FALSE,\n//< Types of Values literal-ops\n//> Global Variables pop-op\n OP_POP,\n//< Global Variables pop-op\n//> Local Variables get-local-op\n OP_GET_LOCAL,\n//< Local Variables get-local-op\n//> Local Variables set-local-op\n OP_SET_LOCAL,\n//< Local Variables set-local-op\n//> Global Variables get-global-op\n OP_GET_GLOBAL,\n//< Global Variables get-global-op\n//> Global Variables define-global-op\n OP_DEFINE_GLOBAL,\n//< Global Variables define-global-op\n//> Global Variables set-global-op\n OP_SET_GLOBAL,\n//< Global Variables set-global-op\n//> Closures upvalue-ops\n OP_GET_UPVALUE,\n OP_SET_UPVALUE,\n//< Closures upvalue-ops\n//> Classes and Instances property-ops\n OP_GET_PROPERTY,\n OP_SET_PROPERTY,\n//< Classes and Instances property-ops\n//> Superclasses get-super-op\n OP_GET_SUPER,\n//< Superclasses get-super-op\n//> Types of Values comparison-ops\n OP_EQUAL,\n OP_GREATER,\n OP_LESS,\n//< Types of Values comparison-ops\n//> A Virtual Machine binary-ops\n OP_ADD,\n OP_SUBTRACT,\n OP_MULTIPLY,\n OP_DIVIDE,\n//> Types of Values not-op\n OP_NOT,\n//< Types of Values not-op\n//< A Virtual Machine binary-ops\n//> A Virtual Machine negate-op\n OP_NEGATE,\n//< A Virtual Machine negate-op\n//> Global Variables op-print\n OP_PRINT,\n//< Global Variables op-print\n//> Jumping Back and Forth jump-op\n OP_JUMP,\n//< Jumping Back and Forth jump-op\n//> Jumping Back and Forth jump-if-false-op\n OP_JUMP_IF_FALSE,\n//< Jumping Back and Forth jump-if-false-op\n//> Jumping Back and Forth loop-op\n OP_LOOP,\n//< Jumping Back and Forth loop-op\n//> Calls and Functions op-call\n OP_CALL,\n//< Calls and Functions op-call\n//> Methods and Initializers invoke-op\n OP_INVOKE,\n//< Methods and Initializers invoke-op\n//> Superclasses super-invoke-op\n OP_SUPER_INVOKE,\n//< Superclasses super-invoke-op\n//> Closures closure-op\n OP_CLOSURE,\n//< Closures closure-op\n//> Closures close-upvalue-op\n OP_CLOSE_UPVALUE,\n//< Closures close-upvalue-op\n OP_RETURN,\n//> Classes and Instances class-op\n OP_CLASS,\n//< Classes and Instances class-op\n//> Superclasses inherit-op\n OP_INHERIT,\n//< Superclasses inherit-op\n//> Methods and Initializers method-op\n OP_METHOD\n//< Methods and Initializers method-op\n} OpCode;\n//< op-enum\n//> chunk-struct\n\ntypedef struct {\n//> count-and-capacity\n int count;\n int capacity;\n//< count-and-capacity\n uint8_t* code;\n//> chunk-lines\n int* lines;\n//< chunk-lines\n//> chunk-constants\n ValueArray constants;\n//< chunk-constants\n} Chunk;\n//< chunk-struct\n//> init-chunk-h\n\nvoid initChunk(Chunk* chunk);\n//< init-chunk-h\n//> free-chunk-h\nvoid freeChunk(Chunk* chunk);\n//< free-chunk-h\n/* Chunks of Bytecode write-chunk-h < Chunks of Bytecode write-chunk-with-line-h\nvoid writeChunk(Chunk* chunk, uint8_t byte);\n*/\n//> write-chunk-with-line-h\nvoid writeChunk(Chunk* chunk, uint8_t byte, int line);\n//< write-chunk-with-line-h\n//> add-constant-h\nint addConstant(Chunk* chunk, Value value);\n//< add-constant-h\n\n#endif\n" }, { "path": "c/clox.xcodeproj/project.pbxproj", "content": "// !$*UTF8*$!\n{\n\tarchiveVersion = 1;\n\tclasses = {\n\t};\n\tobjectVersion = 46;\n\tobjects = {\n\n/* Begin PBXBuildFile section */\n\t\t2905EA1B1CAC1C3900E258E5 /* memory.c in Sources */ = {isa = PBXBuildFile; fileRef = 2905EA191CAC1C3900E258E5 /* memory.c */; };\n\t\t293173A51D03628E0028CBCC /* chunk.c in Sources */ = {isa = PBXBuildFile; fileRef = 293173A31D03628E0028CBCC /* chunk.c */; };\n\t\t293173A81D0378530028CBCC /* value.c in Sources */ = {isa = PBXBuildFile; fileRef = 293173A61D0378530028CBCC /* value.c */; };\n\t\t2940770F1C8368CF0067320B /* vm.c in Sources */ = {isa = PBXBuildFile; fileRef = 2940770D1C8368CF0067320B /* vm.c */; };\n\t\t294077121C8369BC0067320B /* compiler.c in Sources */ = {isa = PBXBuildFile; fileRef = 294077101C8369BC0067320B /* compiler.c */; };\n\t\t296041FF1C5DCCD0007310F9 /* scanner.c in Sources */ = {isa = PBXBuildFile; fileRef = 296041FE1C5DCCD0007310F9 /* scanner.c */; };\n\t\t29815E3F1C5DCC3A004A67D8 /* main.c in Sources */ = {isa = PBXBuildFile; fileRef = 29815E3E1C5DCC3A004A67D8 /* main.c */; };\n\t\t2984DBA21C83FD540075BAC3 /* object.c in Sources */ = {isa = PBXBuildFile; fileRef = 2984DBA01C83FD540075BAC3 /* object.c */; };\n\t\t29C6CA711C85EBE6009617A9 /* debug.c in Sources */ = {isa = PBXBuildFile; fileRef = 29C6CA6F1C85EBE6009617A9 /* debug.c */; };\n\t\t29CD6FB01CB6A3430005D92B /* table.c in Sources */ = {isa = PBXBuildFile; fileRef = 29CD6FAE1CB6A3430005D92B /* table.c */; };\n/* End PBXBuildFile section */\n\n/* Begin PBXCopyFilesBuildPhase section */\n\t\t292D23761E10F6590044C66E /* CopyFiles */ = {\n\t\t\tisa = PBXCopyFilesBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tdstPath = /usr/share/man/man1/;\n\t\t\tdstSubfolderSpec = 0;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 1;\n\t\t};\n\t\t292D239F1E10F6C30044C66E /* CopyFiles */ = {\n\t\t\tisa = PBXCopyFilesBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tdstPath = /usr/share/man/man1/;\n\t\t\tdstSubfolderSpec = 0;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 1;\n\t\t};\n\t\t292D23BC1E10F6E70044C66E /* CopyFiles */ = {\n\t\t\tisa = PBXCopyFilesBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tdstPath = /usr/share/man/man1/;\n\t\t\tdstSubfolderSpec = 0;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 1;\n\t\t};\n\t\t29815E321C5DCBF7004A67D8 /* CopyFiles */ = {\n\t\t\tisa = PBXCopyFilesBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tdstPath = /usr/share/man/man1/;\n\t\t\tdstSubfolderSpec = 0;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 1;\n\t\t};\n/* End PBXCopyFilesBuildPhase section */\n\n/* Begin PBXFileReference section */\n\t\t2905EA191CAC1C3900E258E5 /* memory.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = memory.c; sourceTree = \"\"; };\n\t\t2905EA1A1CAC1C3900E258E5 /* memory.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = memory.h; sourceTree = \"\"; };\n\t\t2905EA1C1CAC1DFB00E258E5 /* common.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = common.h; sourceTree = \"\"; };\n\t\t292D23781E10F6590044C66E /* chap14_chunks */ = {isa = PBXFileReference; explicitFileType = \"compiled.mach-o.executable\"; includeInIndex = 0; path = chap14_chunks; sourceTree = BUILT_PRODUCTS_DIR; };\n\t\t292D23A11E10F6C30044C66E /* chap15_virtual */ = {isa = PBXFileReference; explicitFileType = \"compiled.mach-o.executable\"; includeInIndex = 0; path = chap15_virtual; sourceTree = BUILT_PRODUCTS_DIR; };\n\t\t292D23BE1E10F6E70044C66E /* chap16_scanning */ = {isa = PBXFileReference; explicitFileType = \"compiled.mach-o.executable\"; includeInIndex = 0; path = chap16_scanning; sourceTree = BUILT_PRODUCTS_DIR; };\n\t\t293173A31D03628E0028CBCC /* chunk.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = chunk.c; sourceTree = \"\"; };\n\t\t293173A41D03628E0028CBCC /* chunk.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = chunk.h; sourceTree = \"\"; };\n\t\t293173A61D0378530028CBCC /* value.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = value.c; sourceTree = \"\"; };\n\t\t293173A71D0378530028CBCC /* value.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = value.h; sourceTree = \"\"; };\n\t\t2940770D1C8368CF0067320B /* vm.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = vm.c; sourceTree = \"\"; };\n\t\t2940770E1C8368CF0067320B /* vm.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = vm.h; sourceTree = \"\"; };\n\t\t294077101C8369BC0067320B /* compiler.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = compiler.c; sourceTree = \"\"; };\n\t\t294077111C8369BC0067320B /* compiler.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = compiler.h; sourceTree = \"\"; };\n\t\t296041FE1C5DCCD0007310F9 /* scanner.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = scanner.c; sourceTree = \"\"; };\n\t\t29815E341C5DCBF7004A67D8 /* clox */ = {isa = PBXFileReference; explicitFileType = \"compiled.mach-o.executable\"; includeInIndex = 0; path = clox; sourceTree = BUILT_PRODUCTS_DIR; };\n\t\t29815E3E1C5DCC3A004A67D8 /* main.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = main.c; sourceTree = \"\"; };\n\t\t29815E401C5DCCAC004A67D8 /* scanner.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = scanner.h; sourceTree = \"\"; };\n\t\t2984DBA01C83FD540075BAC3 /* object.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = object.c; sourceTree = \"\"; };\n\t\t2984DBA11C83FD540075BAC3 /* object.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = object.h; sourceTree = \"\"; };\n\t\t29C6CA6F1C85EBE6009617A9 /* debug.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = debug.c; sourceTree = \"\"; };\n\t\t29C6CA701C85EBE6009617A9 /* debug.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = debug.h; sourceTree = \"\"; };\n\t\t29CD6FAE1CB6A3430005D92B /* table.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = table.c; sourceTree = \"\"; };\n\t\t29CD6FAF1CB6A3430005D92B /* table.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = table.h; sourceTree = \"\"; };\n/* End PBXFileReference section */\n\n/* Begin PBXFrameworksBuildPhase section */\n\t\t292D23751E10F6590044C66E /* Frameworks */ = {\n\t\t\tisa = PBXFrameworksBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 0;\n\t\t};\n\t\t292D239E1E10F6C30044C66E /* Frameworks */ = {\n\t\t\tisa = PBXFrameworksBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 0;\n\t\t};\n\t\t292D23BB1E10F6E70044C66E /* Frameworks */ = {\n\t\t\tisa = PBXFrameworksBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 0;\n\t\t};\n\t\t29815E311C5DCBF7004A67D8 /* Frameworks */ = {\n\t\t\tisa = PBXFrameworksBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 0;\n\t\t};\n/* End PBXFrameworksBuildPhase section */\n\n/* Begin PBXGroup section */\n\t\t29815E2B1C5DCBF7004A67D8 = {\n\t\t\tisa = PBXGroup;\n\t\t\tchildren = (\n\t\t\t\t293173A41D03628E0028CBCC /* chunk.h */,\n\t\t\t\t293173A31D03628E0028CBCC /* chunk.c */,\n\t\t\t\t2905EA1C1CAC1DFB00E258E5 /* common.h */,\n\t\t\t\t294077111C8369BC0067320B /* compiler.h */,\n\t\t\t\t294077101C8369BC0067320B /* compiler.c */,\n\t\t\t\t29C6CA701C85EBE6009617A9 /* debug.h */,\n\t\t\t\t29C6CA6F1C85EBE6009617A9 /* debug.c */,\n\t\t\t\t29815E3E1C5DCC3A004A67D8 /* main.c */,\n\t\t\t\t2905EA1A1CAC1C3900E258E5 /* memory.h */,\n\t\t\t\t2905EA191CAC1C3900E258E5 /* memory.c */,\n\t\t\t\t2984DBA11C83FD540075BAC3 /* object.h */,\n\t\t\t\t2984DBA01C83FD540075BAC3 /* object.c */,\n\t\t\t\t29815E401C5DCCAC004A67D8 /* scanner.h */,\n\t\t\t\t296041FE1C5DCCD0007310F9 /* scanner.c */,\n\t\t\t\t29CD6FAF1CB6A3430005D92B /* table.h */,\n\t\t\t\t29CD6FAE1CB6A3430005D92B /* table.c */,\n\t\t\t\t293173A71D0378530028CBCC /* value.h */,\n\t\t\t\t293173A61D0378530028CBCC /* value.c */,\n\t\t\t\t2940770E1C8368CF0067320B /* vm.h */,\n\t\t\t\t2940770D1C8368CF0067320B /* vm.c */,\n\t\t\t\t29815E351C5DCBF7004A67D8 /* Products */,\n\t\t\t);\n\t\t\tsourceTree = \"\";\n\t\t};\n\t\t29815E351C5DCBF7004A67D8 /* Products */ = {\n\t\t\tisa = PBXGroup;\n\t\t\tchildren = (\n\t\t\t\t29815E341C5DCBF7004A67D8 /* clox */,\n\t\t\t\t292D23781E10F6590044C66E /* chap14_chunks */,\n\t\t\t\t292D23A11E10F6C30044C66E /* chap15_virtual */,\n\t\t\t\t292D23BE1E10F6E70044C66E /* chap16_scanning */,\n\t\t\t);\n\t\t\tname = Products;\n\t\t\tsourceTree = \"\";\n\t\t};\n/* End PBXGroup section */\n\n/* Begin PBXNativeTarget section */\n\t\t292D23771E10F6590044C66E /* chap14_chunks */ = {\n\t\t\tisa = PBXNativeTarget;\n\t\t\tbuildConfigurationList = 292D237C1E10F6590044C66E /* Build configuration list for PBXNativeTarget \"chap14_chunks\" */;\n\t\t\tbuildPhases = (\n\t\t\t\t292D23741E10F6590044C66E /* Sources */,\n\t\t\t\t292D23751E10F6590044C66E /* Frameworks */,\n\t\t\t\t292D23761E10F6590044C66E /* CopyFiles */,\n\t\t\t);\n\t\t\tbuildRules = (\n\t\t\t);\n\t\t\tdependencies = (\n\t\t\t);\n\t\t\tname = chap14_chunks;\n\t\t\tproductName = chap14_chunks;\n\t\t\tproductReference = 292D23781E10F6590044C66E /* chap14_chunks */;\n\t\t\tproductType = \"com.apple.product-type.tool\";\n\t\t};\n\t\t292D23A01E10F6C30044C66E /* chap15_virtual */ = {\n\t\t\tisa = PBXNativeTarget;\n\t\t\tbuildConfigurationList = 292D23A51E10F6C30044C66E /* Build configuration list for PBXNativeTarget \"chap15_virtual\" */;\n\t\t\tbuildPhases = (\n\t\t\t\t292D239D1E10F6C30044C66E /* Sources */,\n\t\t\t\t292D239E1E10F6C30044C66E /* Frameworks */,\n\t\t\t\t292D239F1E10F6C30044C66E /* CopyFiles */,\n\t\t\t);\n\t\t\tbuildRules = (\n\t\t\t);\n\t\t\tdependencies = (\n\t\t\t);\n\t\t\tname = chap15_virtual;\n\t\t\tproductName = chap15_virtual;\n\t\t\tproductReference = 292D23A11E10F6C30044C66E /* chap15_virtual */;\n\t\t\tproductType = \"com.apple.product-type.tool\";\n\t\t};\n\t\t292D23BD1E10F6E70044C66E /* chap16_scanning */ = {\n\t\t\tisa = PBXNativeTarget;\n\t\t\tbuildConfigurationList = 292D23C21E10F6E70044C66E /* Build configuration list for PBXNativeTarget \"chap16_scanning\" */;\n\t\t\tbuildPhases = (\n\t\t\t\t292D23BA1E10F6E70044C66E /* Sources */,\n\t\t\t\t292D23BB1E10F6E70044C66E /* Frameworks */,\n\t\t\t\t292D23BC1E10F6E70044C66E /* CopyFiles */,\n\t\t\t);\n\t\t\tbuildRules = (\n\t\t\t);\n\t\t\tdependencies = (\n\t\t\t);\n\t\t\tname = chap16_scanning;\n\t\t\tproductName = chap16_scanning;\n\t\t\tproductReference = 292D23BE1E10F6E70044C66E /* chap16_scanning */;\n\t\t\tproductType = \"com.apple.product-type.tool\";\n\t\t};\n\t\t29815E331C5DCBF7004A67D8 /* clox */ = {\n\t\t\tisa = PBXNativeTarget;\n\t\t\tbuildConfigurationList = 29815E3B1C5DCBF7004A67D8 /* Build configuration list for PBXNativeTarget \"clox\" */;\n\t\t\tbuildPhases = (\n\t\t\t\t29815E301C5DCBF7004A67D8 /* Sources */,\n\t\t\t\t29815E311C5DCBF7004A67D8 /* Frameworks */,\n\t\t\t\t29815E321C5DCBF7004A67D8 /* CopyFiles */,\n\t\t\t);\n\t\t\tbuildRules = (\n\t\t\t);\n\t\t\tdependencies = (\n\t\t\t);\n\t\t\tname = clox;\n\t\t\tproductName = cvox;\n\t\t\tproductReference = 29815E341C5DCBF7004A67D8 /* clox */;\n\t\t\tproductType = \"com.apple.product-type.tool\";\n\t\t};\n/* End PBXNativeTarget section */\n\n/* Begin PBXProject section */\n\t\t29815E2C1C5DCBF7004A67D8 /* Project object */ = {\n\t\t\tisa = PBXProject;\n\t\t\tattributes = {\n\t\t\t\tLastUpgradeCheck = 0830;\n\t\t\t\tORGANIZATIONNAME = \"Robert Nystrom\";\n\t\t\t\tTargetAttributes = {\n\t\t\t\t\t292D23771E10F6590044C66E = {\n\t\t\t\t\t\tCreatedOnToolsVersion = 8.1;\n\t\t\t\t\t\tProvisioningStyle = Automatic;\n\t\t\t\t\t};\n\t\t\t\t\t292D23A01E10F6C30044C66E = {\n\t\t\t\t\t\tCreatedOnToolsVersion = 8.1;\n\t\t\t\t\t\tProvisioningStyle = Automatic;\n\t\t\t\t\t};\n\t\t\t\t\t292D23BD1E10F6E70044C66E = {\n\t\t\t\t\t\tCreatedOnToolsVersion = 8.1;\n\t\t\t\t\t\tProvisioningStyle = Automatic;\n\t\t\t\t\t};\n\t\t\t\t\t29815E331C5DCBF7004A67D8 = {\n\t\t\t\t\t\tCreatedOnToolsVersion = 6.4;\n\t\t\t\t\t};\n\t\t\t\t};\n\t\t\t};\n\t\t\tbuildConfigurationList = 29815E2F1C5DCBF7004A67D8 /* Build configuration list for PBXProject \"clox\" */;\n\t\t\tcompatibilityVersion = \"Xcode 3.2\";\n\t\t\tdevelopmentRegion = English;\n\t\t\thasScannedForEncodings = 0;\n\t\t\tknownRegions = (\n\t\t\t\tEnglish,\n\t\t\t\ten,\n\t\t\t);\n\t\t\tmainGroup = 29815E2B1C5DCBF7004A67D8;\n\t\t\tproductRefGroup = 29815E351C5DCBF7004A67D8 /* Products */;\n\t\t\tprojectDirPath = \"\";\n\t\t\tprojectRoot = \"\";\n\t\t\ttargets = (\n\t\t\t\t29815E331C5DCBF7004A67D8 /* clox */,\n\t\t\t\t292D23771E10F6590044C66E /* chap14_chunks */,\n\t\t\t\t292D23A01E10F6C30044C66E /* chap15_virtual */,\n\t\t\t\t292D23BD1E10F6E70044C66E /* chap16_scanning */,\n\t\t\t);\n\t\t};\n/* End PBXProject section */\n\n/* Begin PBXSourcesBuildPhase section */\n\t\t292D23741E10F6590044C66E /* Sources */ = {\n\t\t\tisa = PBXSourcesBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 0;\n\t\t};\n\t\t292D239D1E10F6C30044C66E /* Sources */ = {\n\t\t\tisa = PBXSourcesBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 0;\n\t\t};\n\t\t292D23BA1E10F6E70044C66E /* Sources */ = {\n\t\t\tisa = PBXSourcesBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tfiles = (\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 0;\n\t\t};\n\t\t29815E301C5DCBF7004A67D8 /* Sources */ = {\n\t\t\tisa = PBXSourcesBuildPhase;\n\t\t\tbuildActionMask = 2147483647;\n\t\t\tfiles = (\n\t\t\t\t293173A51D03628E0028CBCC /* chunk.c in Sources */,\n\t\t\t\t29CD6FB01CB6A3430005D92B /* table.c in Sources */,\n\t\t\t\t2905EA1B1CAC1C3900E258E5 /* memory.c in Sources */,\n\t\t\t\t2984DBA21C83FD540075BAC3 /* object.c in Sources */,\n\t\t\t\t296041FF1C5DCCD0007310F9 /* scanner.c in Sources */,\n\t\t\t\t293173A81D0378530028CBCC /* value.c in Sources */,\n\t\t\t\t2940770F1C8368CF0067320B /* vm.c in Sources */,\n\t\t\t\t29C6CA711C85EBE6009617A9 /* debug.c in Sources */,\n\t\t\t\t294077121C8369BC0067320B /* compiler.c in Sources */,\n\t\t\t\t29815E3F1C5DCC3A004A67D8 /* main.c in Sources */,\n\t\t\t);\n\t\t\trunOnlyForDeploymentPostprocessing = 0;\n\t\t};\n/* End PBXSourcesBuildPhase section */\n\n/* Begin XCBuildConfiguration section */\n\t\t292D237D1E10F6590044C66E /* Debug */ = {\n\t\t\tisa = XCBuildConfiguration;\n\t\t\tbuildSettings = {\n\t\t\t\tCLANG_ANALYZER_NONNULL = YES;\n\t\t\t\tCLANG_WARN_DOCUMENTATION_COMMENTS = YES;\n\t\t\t\tCLANG_WARN_INFINITE_RECURSION = YES;\n\t\t\t\tCLANG_WARN_SUSPICIOUS_MOVES = YES;\n\t\t\t\tCODE_SIGN_IDENTITY = \"-\";\n\t\t\t\tENABLE_TESTABILITY = YES;\n\t\t\t\tMACOSX_DEPLOYMENT_TARGET = 10.12;\n\t\t\t\tPRODUCT_NAME = \"$(TARGET_NAME)\";\n\t\t\t};\n\t\t\tname = Debug;\n\t\t};\n\t\t292D237E1E10F6590044C66E /* Release */ = {\n\t\t\tisa = XCBuildConfiguration;\n\t\t\tbuildSettings = {\n\t\t\t\tCLANG_ANALYZER_NONNULL = YES;\n\t\t\t\tCLANG_WARN_DOCUMENTATION_COMMENTS = YES;\n\t\t\t\tCLANG_WARN_INFINITE_RECURSION = YES;\n\t\t\t\tCLANG_WARN_SUSPICIOUS_MOVES = YES;\n\t\t\t\tCODE_SIGN_IDENTITY = \"-\";\n\t\t\t\tMACOSX_DEPLOYMENT_TARGET = 10.12;\n\t\t\t\tPRODUCT_NAME = \"$(TARGET_NAME)\";\n\t\t\t};\n\t\t\tname = Release;\n\t\t};\n\t\t292D23A61E10F6C30044C66E /* Debug */ = {\n\t\t\tisa = XCBuildConfiguration;\n\t\t\tbuildSettings = {\n\t\t\t\tCLANG_ANALYZER_NONNULL = YES;\n\t\t\t\tCLANG_WARN_DOCUMENTATION_COMMENTS = YES;\n\t\t\t\tCLANG_WARN_INFINITE_RECURSION = YES;\n\t\t\t\tCLANG_WARN_SUSPICIOUS_MOVES = YES;\n\t\t\t\tCODE_SIGN_IDENTITY = \"-\";\n\t\t\t\tENABLE_TESTABILITY = YES;\n\t\t\t\tMACOSX_DEPLOYMENT_TARGET = 10.12;\n\t\t\t\tPRODUCT_NAME = \"$(TARGET_NAME)\";\n\t\t\t};\n\t\t\tname = Debug;\n\t\t};\n\t\t292D23A71E10F6C30044C66E /* Release */ = {\n\t\t\tisa = XCBuildConfiguration;\n\t\t\tbuildSettings = {\n\t\t\t\tCLANG_ANALYZER_NONNULL = YES;\n\t\t\t\tCLANG_WARN_DOCUMENTATION_COMMENTS = YES;\n\t\t\t\tCLANG_WARN_INFINITE_RECURSION = YES;\n\t\t\t\tCLANG_WARN_SUSPICIOUS_MOVES = YES;\n\t\t\t\tCODE_SIGN_IDENTITY = \"-\";\n\t\t\t\tMACOSX_DEPLOYMENT_TARGET = 10.12;\n\t\t\t\tPRODUCT_NAME = \"$(TARGET_NAME)\";\n\t\t\t};\n\t\t\tname = Release;\n\t\t};\n\t\t292D23C31E10F6E70044C66E /* Debug */ = {\n\t\t\tisa = XCBuildConfiguration;\n\t\t\tbuildSettings = {\n\t\t\t\tCLANG_ANALYZER_NONNULL = YES;\n\t\t\t\tCLANG_WARN_DOCUMENTATION_COMMENTS = YES;\n\t\t\t\tCLANG_WARN_INFINITE_RECURSION = YES;\n\t\t\t\tCLANG_WARN_SUSPICIOUS_MOVES = YES;\n\t\t\t\tCODE_SIGN_IDENTITY = \"-\";\n\t\t\t\tENABLE_TESTABILITY = YES;\n\t\t\t\tMACOSX_DEPLOYMENT_TARGET = 10.12;\n\t\t\t\tPRODUCT_NAME = \"$(TARGET_NAME)\";\n\t\t\t};\n\t\t\tname = Debug;\n\t\t};\n\t\t292D23C41E10F6E70044C66E /* Release */ = {\n\t\t\tisa = XCBuildConfiguration;\n\t\t\tbuildSettings = {\n\t\t\t\tCLANG_ANALYZER_NONNULL = YES;\n\t\t\t\tCLANG_WARN_DOCUMENTATION_COMMENTS = YES;\n\t\t\t\tCLANG_WARN_INFINITE_RECURSION = YES;\n\t\t\t\tCLANG_WARN_SUSPICIOUS_MOVES = YES;\n\t\t\t\tCODE_SIGN_IDENTITY = \"-\";\n\t\t\t\tMACOSX_DEPLOYMENT_TARGET = 10.12;\n\t\t\t\tPRODUCT_NAME = \"$(TARGET_NAME)\";\n\t\t\t};\n\t\t\tname = Release;\n\t\t};\n\t\t29815E391C5DCBF7004A67D8 /* Debug */ = {\n\t\t\tisa = XCBuildConfiguration;\n\t\t\tbuildSettings = {\n\t\t\t\tALWAYS_SEARCH_USER_PATHS = NO;\n\t\t\t\tCLANG_CXX_LANGUAGE_STANDARD = \"gnu++0x\";\n\t\t\t\tCLANG_CXX_LIBRARY = \"libc++\";\n\t\t\t\tCLANG_ENABLE_MODULES = YES;\n\t\t\t\tCLANG_ENABLE_OBJC_ARC = YES;\n\t\t\t\tCLANG_WARN_BOOL_CONVERSION = YES;\n\t\t\t\tCLANG_WARN_CONSTANT_CONVERSION = YES;\n\t\t\t\tCLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR;\n\t\t\t\tCLANG_WARN_EMPTY_BODY = YES;\n\t\t\t\tCLANG_WARN_ENUM_CONVERSION = YES;\n\t\t\t\tCLANG_WARN_INFINITE_RECURSION = YES;\n\t\t\t\tCLANG_WARN_INT_CONVERSION = YES;\n\t\t\t\tCLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR;\n\t\t\t\tCLANG_WARN_SUSPICIOUS_MOVE = YES;\n\t\t\t\tCLANG_WARN_UNREACHABLE_CODE = YES;\n\t\t\t\tCLANG_WARN__DUPLICATE_METHOD_MATCH = YES;\n\t\t\t\tCOPY_PHASE_STRIP = NO;\n\t\t\t\tDEBUG_INFORMATION_FORMAT = dwarf;\n\t\t\t\tENABLE_STRICT_OBJC_MSGSEND = YES;\n\t\t\t\tENABLE_TESTABILITY = YES;\n\t\t\t\tGCC_C_LANGUAGE_STANDARD = gnu99;\n\t\t\t\tGCC_DYNAMIC_NO_PIC = NO;\n\t\t\t\tGCC_NO_COMMON_BLOCKS = YES;\n\t\t\t\tGCC_OPTIMIZATION_LEVEL = 0;\n\t\t\t\tGCC_PREPROCESSOR_DEFINITIONS = (\n\t\t\t\t\t\"DEBUG=1\",\n\t\t\t\t\t\"$(inherited)\",\n\t\t\t\t);\n\t\t\t\tGCC_SYMBOLS_PRIVATE_EXTERN = NO;\n\t\t\t\tGCC_WARN_64_TO_32_BIT_CONVERSION = YES;\n\t\t\t\tGCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR;\n\t\t\t\tGCC_WARN_UNDECLARED_SELECTOR = YES;\n\t\t\t\tGCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE;\n\t\t\t\tGCC_WARN_UNUSED_FUNCTION = YES;\n\t\t\t\tGCC_WARN_UNUSED_VARIABLE = YES;\n\t\t\t\tMACOSX_DEPLOYMENT_TARGET = 10.11;\n\t\t\t\tMTL_ENABLE_DEBUG_INFO = YES;\n\t\t\t\tONLY_ACTIVE_ARCH = YES;\n\t\t\t\tSDKROOT = macosx;\n\t\t\t\tWARNING_CFLAGS = \"-Wno-gnu-label-as-value\";\n\t\t\t};\n\t\t\tname = Debug;\n\t\t};\n\t\t29815E3A1C5DCBF7004A67D8 /* Release */ = {\n\t\t\tisa = XCBuildConfiguration;\n\t\t\tbuildSettings = {\n\t\t\t\tALWAYS_SEARCH_USER_PATHS = NO;\n\t\t\t\tCLANG_CXX_LANGUAGE_STANDARD = \"gnu++0x\";\n\t\t\t\tCLANG_CXX_LIBRARY = \"libc++\";\n\t\t\t\tCLANG_ENABLE_MODULES = YES;\n\t\t\t\tCLANG_ENABLE_OBJC_ARC = YES;\n\t\t\t\tCLANG_WARN_BOOL_CONVERSION = YES;\n\t\t\t\tCLANG_WARN_CONSTANT_CONVERSION = YES;\n\t\t\t\tCLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR;\n\t\t\t\tCLANG_WARN_EMPTY_BODY = YES;\n\t\t\t\tCLANG_WARN_ENUM_CONVERSION = YES;\n\t\t\t\tCLANG_WARN_INFINITE_RECURSION = YES;\n\t\t\t\tCLANG_WARN_INT_CONVERSION = YES;\n\t\t\t\tCLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR;\n\t\t\t\tCLANG_WARN_SUSPICIOUS_MOVE = YES;\n\t\t\t\tCLANG_WARN_UNREACHABLE_CODE = YES;\n\t\t\t\tCLANG_WARN__DUPLICATE_METHOD_MATCH = YES;\n\t\t\t\tCOPY_PHASE_STRIP = NO;\n\t\t\t\tDEBUG_INFORMATION_FORMAT = \"dwarf-with-dsym\";\n\t\t\t\tENABLE_NS_ASSERTIONS = NO;\n\t\t\t\tENABLE_STRICT_OBJC_MSGSEND = YES;\n\t\t\t\tGCC_C_LANGUAGE_STANDARD = gnu99;\n\t\t\t\tGCC_NO_COMMON_BLOCKS = YES;\n\t\t\t\tGCC_OPTIMIZATION_LEVEL = 3;\n\t\t\t\tGCC_WARN_64_TO_32_BIT_CONVERSION = YES;\n\t\t\t\tGCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR;\n\t\t\t\tGCC_WARN_UNDECLARED_SELECTOR = YES;\n\t\t\t\tGCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE;\n\t\t\t\tGCC_WARN_UNUSED_FUNCTION = YES;\n\t\t\t\tGCC_WARN_UNUSED_VARIABLE = YES;\n\t\t\t\tMACOSX_DEPLOYMENT_TARGET = 10.11;\n\t\t\t\tMTL_ENABLE_DEBUG_INFO = NO;\n\t\t\t\tSDKROOT = macosx;\n\t\t\t\tWARNING_CFLAGS = \"-Wno-gnu-label-as-value\";\n\t\t\t};\n\t\t\tname = Release;\n\t\t};\n\t\t29815E3C1C5DCBF7004A67D8 /* Debug */ = {\n\t\t\tisa = XCBuildConfiguration;\n\t\t\tbuildSettings = {\n\t\t\t\tCLANG_WARN_SUSPICIOUS_IMPLICIT_CONVERSION = YES;\n\t\t\t\tGCC_C_LANGUAGE_STANDARD = c99;\n\t\t\t\tGCC_TREAT_IMPLICIT_FUNCTION_DECLARATIONS_AS_ERRORS = YES;\n\t\t\t\tGCC_TREAT_INCOMPATIBLE_POINTER_TYPE_WARNINGS_AS_ERRORS = YES;\n\t\t\t\tGCC_TREAT_WARNINGS_AS_ERRORS = YES;\n\t\t\t\tGCC_WARN_ABOUT_MISSING_FIELD_INITIALIZERS = YES;\n\t\t\t\tGCC_WARN_PEDANTIC = YES;\n\t\t\t\tGCC_WARN_SHADOW = YES;\n\t\t\t\tMACOSX_DEPLOYMENT_TARGET = 10.14;\n\t\t\t\tPRODUCT_NAME = \"$(TARGET_NAME)\";\n\t\t\t};\n\t\t\tname = Debug;\n\t\t};\n\t\t29815E3D1C5DCBF7004A67D8 /* Release */ = {\n\t\t\tisa = XCBuildConfiguration;\n\t\t\tbuildSettings = {\n\t\t\t\tCLANG_WARN_SUSPICIOUS_IMPLICIT_CONVERSION = YES;\n\t\t\t\tGCC_C_LANGUAGE_STANDARD = c99;\n\t\t\t\tGCC_TREAT_IMPLICIT_FUNCTION_DECLARATIONS_AS_ERRORS = YES;\n\t\t\t\tGCC_TREAT_INCOMPATIBLE_POINTER_TYPE_WARNINGS_AS_ERRORS = YES;\n\t\t\t\tGCC_TREAT_WARNINGS_AS_ERRORS = YES;\n\t\t\t\tGCC_WARN_ABOUT_MISSING_FIELD_INITIALIZERS = YES;\n\t\t\t\tGCC_WARN_PEDANTIC = YES;\n\t\t\t\tGCC_WARN_SHADOW = YES;\n\t\t\t\tMACOSX_DEPLOYMENT_TARGET = 10.14;\n\t\t\t\tPRODUCT_NAME = \"$(TARGET_NAME)\";\n\t\t\t};\n\t\t\tname = Release;\n\t\t};\n/* End XCBuildConfiguration section */\n\n/* Begin XCConfigurationList section */\n\t\t292D237C1E10F6590044C66E /* Build configuration list for PBXNativeTarget \"chap14_chunks\" */ = {\n\t\t\tisa = XCConfigurationList;\n\t\t\tbuildConfigurations = (\n\t\t\t\t292D237D1E10F6590044C66E /* Debug */,\n\t\t\t\t292D237E1E10F6590044C66E /* Release */,\n\t\t\t);\n\t\t\tdefaultConfigurationIsVisible = 0;\n\t\t\tdefaultConfigurationName = Release;\n\t\t};\n\t\t292D23A51E10F6C30044C66E /* Build configuration list for PBXNativeTarget \"chap15_virtual\" */ = {\n\t\t\tisa = XCConfigurationList;\n\t\t\tbuildConfigurations = (\n\t\t\t\t292D23A61E10F6C30044C66E /* Debug */,\n\t\t\t\t292D23A71E10F6C30044C66E /* Release */,\n\t\t\t);\n\t\t\tdefaultConfigurationIsVisible = 0;\n\t\t\tdefaultConfigurationName = Release;\n\t\t};\n\t\t292D23C21E10F6E70044C66E /* Build configuration list for PBXNativeTarget \"chap16_scanning\" */ = {\n\t\t\tisa = XCConfigurationList;\n\t\t\tbuildConfigurations = (\n\t\t\t\t292D23C31E10F6E70044C66E /* Debug */,\n\t\t\t\t292D23C41E10F6E70044C66E /* Release */,\n\t\t\t);\n\t\t\tdefaultConfigurationIsVisible = 0;\n\t\t\tdefaultConfigurationName = Release;\n\t\t};\n\t\t29815E2F1C5DCBF7004A67D8 /* Build configuration list for PBXProject \"clox\" */ = {\n\t\t\tisa = XCConfigurationList;\n\t\t\tbuildConfigurations = (\n\t\t\t\t29815E391C5DCBF7004A67D8 /* Debug */,\n\t\t\t\t29815E3A1C5DCBF7004A67D8 /* Release */,\n\t\t\t);\n\t\t\tdefaultConfigurationIsVisible = 0;\n\t\t\tdefaultConfigurationName = Release;\n\t\t};\n\t\t29815E3B1C5DCBF7004A67D8 /* Build configuration list for PBXNativeTarget \"clox\" */ = {\n\t\t\tisa = XCConfigurationList;\n\t\t\tbuildConfigurations = (\n\t\t\t\t29815E3C1C5DCBF7004A67D8 /* Debug */,\n\t\t\t\t29815E3D1C5DCBF7004A67D8 /* Release */,\n\t\t\t);\n\t\t\tdefaultConfigurationIsVisible = 0;\n\t\t\tdefaultConfigurationName = Release;\n\t\t};\n/* End XCConfigurationList section */\n\t};\n\trootObject = 29815E2C1C5DCBF7004A67D8 /* Project object */;\n}\n" }, { "path": "c/clox.xcodeproj/project.xcworkspace/contents.xcworkspacedata", "content": "\n\n \n \n\n" }, { "path": "c/clox.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist", "content": "\n\n\n\n\tIDEDidComputeMac32BitWarning\n\t\n\n\n" }, { "path": "c/clox.xcodeproj/project.xcworkspace/xcshareddata/WorkspaceSettings.xcsettings", "content": "\n\n\n\n\tPreviewsEnabled\n\t\n\n\n" }, { "path": "c/clox.xcodeproj/xcshareddata/xcschemes/clox.xcscheme", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "c/common.h", "content": "//> Chunks of Bytecode common-h\n#ifndef clox_common_h\n#define clox_common_h\n\n#include \n#include \n#include \n//> A Virtual Machine define-debug-trace\n\n//> Optimization define-nan-boxing\n#define NAN_BOXING\n//< Optimization define-nan-boxing\n//> Compiling Expressions define-debug-print-code\n#define DEBUG_PRINT_CODE\n//< Compiling Expressions define-debug-print-code\n#define DEBUG_TRACE_EXECUTION\n//< A Virtual Machine define-debug-trace\n//> Garbage Collection define-stress-gc\n\n#define DEBUG_STRESS_GC\n//< Garbage Collection define-stress-gc\n//> Garbage Collection define-log-gc\n#define DEBUG_LOG_GC\n//< Garbage Collection define-log-gc\n//> Local Variables uint8-count\n\n#define UINT8_COUNT (UINT8_MAX + 1)\n//< Local Variables uint8-count\n\n#endif\n//> omit\n// In the book, we show them defined, but for working on them locally,\n// we don't want them to be.\n#undef DEBUG_PRINT_CODE\n#undef DEBUG_TRACE_EXECUTION\n#undef DEBUG_STRESS_GC\n#undef DEBUG_LOG_GC\n//< omit\n" }, { "path": "c/compiler.c", "content": "//> Scanning on Demand compiler-c\n#include \n//> Compiling Expressions compiler-include-stdlib\n#include \n//< Compiling Expressions compiler-include-stdlib\n//> Local Variables compiler-include-string\n#include \n//< Local Variables compiler-include-string\n\n#include \"common.h\"\n#include \"compiler.h\"\n//> Garbage Collection compiler-include-memory\n#include \"memory.h\"\n//< Garbage Collection compiler-include-memory\n#include \"scanner.h\"\n//> Compiling Expressions include-debug\n\n#ifdef DEBUG_PRINT_CODE\n#include \"debug.h\"\n#endif\n//< Compiling Expressions include-debug\n//> Compiling Expressions parser\n\ntypedef struct {\n Token current;\n Token previous;\n//> had-error-field\n bool hadError;\n//< had-error-field\n//> panic-mode-field\n bool panicMode;\n//< panic-mode-field\n} Parser;\n//> precedence\n\ntypedef enum {\n PREC_NONE,\n PREC_ASSIGNMENT, // =\n PREC_OR, // or\n PREC_AND, // and\n PREC_EQUALITY, // == !=\n PREC_COMPARISON, // < > <= >=\n PREC_TERM, // + -\n PREC_FACTOR, // * /\n PREC_UNARY, // ! -\n PREC_CALL, // . ()\n PREC_PRIMARY\n} Precedence;\n//< precedence\n//> parse-fn-type\n\n//< parse-fn-type\n/* Compiling Expressions parse-fn-type < Global Variables parse-fn-type\ntypedef void (*ParseFn)();\n*/\n//> Global Variables parse-fn-type\ntypedef void (*ParseFn)(bool canAssign);\n//< Global Variables parse-fn-type\n//> parse-rule\n\ntypedef struct {\n ParseFn prefix;\n ParseFn infix;\n Precedence precedence;\n} ParseRule;\n//< parse-rule\n//> Local Variables local-struct\n\ntypedef struct {\n Token name;\n int depth;\n//> Closures is-captured-field\n bool isCaptured;\n//< Closures is-captured-field\n} Local;\n//< Local Variables local-struct\n//> Closures upvalue-struct\ntypedef struct {\n uint8_t index;\n bool isLocal;\n} Upvalue;\n//< Closures upvalue-struct\n//> Calls and Functions function-type-enum\ntypedef enum {\n TYPE_FUNCTION,\n//> Methods and Initializers initializer-type-enum\n TYPE_INITIALIZER,\n//< Methods and Initializers initializer-type-enum\n//> Methods and Initializers method-type-enum\n TYPE_METHOD,\n//< Methods and Initializers method-type-enum\n TYPE_SCRIPT\n} FunctionType;\n//< Calls and Functions function-type-enum\n//> Local Variables compiler-struct\n\n/* Local Variables compiler-struct < Calls and Functions enclosing-field\ntypedef struct {\n*/\n//> Calls and Functions enclosing-field\ntypedef struct Compiler {\n struct Compiler* enclosing;\n//< Calls and Functions enclosing-field\n//> Calls and Functions function-fields\n ObjFunction* function;\n FunctionType type;\n\n//< Calls and Functions function-fields\n Local locals[UINT8_COUNT];\n int localCount;\n//> Closures upvalues-array\n Upvalue upvalues[UINT8_COUNT];\n//< Closures upvalues-array\n int scopeDepth;\n} Compiler;\n//< Local Variables compiler-struct\n//> Methods and Initializers class-compiler-struct\n\ntypedef struct ClassCompiler {\n struct ClassCompiler* enclosing;\n//> Superclasses has-superclass\n bool hasSuperclass;\n//< Superclasses has-superclass\n} ClassCompiler;\n//< Methods and Initializers class-compiler-struct\n\nParser parser;\n//< Compiling Expressions parser\n//> Local Variables current-compiler\nCompiler* current = NULL;\n//< Local Variables current-compiler\n//> Methods and Initializers current-class\nClassCompiler* currentClass = NULL;\n//< Methods and Initializers current-class\n//> Compiling Expressions compiling-chunk\n/* Compiling Expressions compiling-chunk < Calls and Functions current-chunk\nChunk* compilingChunk;\n\nstatic Chunk* currentChunk() {\n return compilingChunk;\n}\n*/\n//> Calls and Functions current-chunk\n\nstatic Chunk* currentChunk() {\n return ¤t->function->chunk;\n}\n//< Calls and Functions current-chunk\n\n//< Compiling Expressions compiling-chunk\n//> Compiling Expressions error-at\nstatic void errorAt(Token* token, const char* message) {\n//> check-panic-mode\n if (parser.panicMode) return;\n//< check-panic-mode\n//> set-panic-mode\n parser.panicMode = true;\n//< set-panic-mode\n fprintf(stderr, \"[line %d] Error\", token->line);\n\n if (token->type == TOKEN_EOF) {\n fprintf(stderr, \" at end\");\n } else if (token->type == TOKEN_ERROR) {\n // Nothing.\n } else {\n fprintf(stderr, \" at '%.*s'\", token->length, token->start);\n }\n\n fprintf(stderr, \": %s\\n\", message);\n parser.hadError = true;\n}\n//< Compiling Expressions error-at\n//> Compiling Expressions error\nstatic void error(const char* message) {\n errorAt(&parser.previous, message);\n}\n//< Compiling Expressions error\n//> Compiling Expressions error-at-current\nstatic void errorAtCurrent(const char* message) {\n errorAt(&parser.current, message);\n}\n//< Compiling Expressions error-at-current\n//> Compiling Expressions advance\n\nstatic void advance() {\n parser.previous = parser.current;\n\n for (;;) {\n parser.current = scanToken();\n if (parser.current.type != TOKEN_ERROR) break;\n\n errorAtCurrent(parser.current.start);\n }\n}\n//< Compiling Expressions advance\n//> Compiling Expressions consume\nstatic void consume(TokenType type, const char* message) {\n if (parser.current.type == type) {\n advance();\n return;\n }\n\n errorAtCurrent(message);\n}\n//< Compiling Expressions consume\n//> Global Variables check\nstatic bool check(TokenType type) {\n return parser.current.type == type;\n}\n//< Global Variables check\n//> Global Variables match\nstatic bool match(TokenType type) {\n if (!check(type)) return false;\n advance();\n return true;\n}\n//< Global Variables match\n//> Compiling Expressions emit-byte\nstatic void emitByte(uint8_t byte) {\n writeChunk(currentChunk(), byte, parser.previous.line);\n}\n//< Compiling Expressions emit-byte\n//> Compiling Expressions emit-bytes\nstatic void emitBytes(uint8_t byte1, uint8_t byte2) {\n emitByte(byte1);\n emitByte(byte2);\n}\n//< Compiling Expressions emit-bytes\n//> Jumping Back and Forth emit-loop\nstatic void emitLoop(int loopStart) {\n emitByte(OP_LOOP);\n\n int offset = currentChunk()->count - loopStart + 2;\n if (offset > UINT16_MAX) error(\"Loop body too large.\");\n\n emitByte((offset >> 8) & 0xff);\n emitByte(offset & 0xff);\n}\n//< Jumping Back and Forth emit-loop\n//> Jumping Back and Forth emit-jump\nstatic int emitJump(uint8_t instruction) {\n emitByte(instruction);\n emitByte(0xff);\n emitByte(0xff);\n return currentChunk()->count - 2;\n}\n//< Jumping Back and Forth emit-jump\n//> Compiling Expressions emit-return\nstatic void emitReturn() {\n/* Calls and Functions return-nil < Methods and Initializers return-this\n emitByte(OP_NIL);\n*/\n//> Methods and Initializers return-this\n if (current->type == TYPE_INITIALIZER) {\n emitBytes(OP_GET_LOCAL, 0);\n } else {\n emitByte(OP_NIL);\n }\n\n//< Methods and Initializers return-this\n emitByte(OP_RETURN);\n}\n//< Compiling Expressions emit-return\n//> Compiling Expressions make-constant\nstatic uint8_t makeConstant(Value value) {\n int constant = addConstant(currentChunk(), value);\n if (constant > UINT8_MAX) {\n error(\"Too many constants in one chunk.\");\n return 0;\n }\n\n return (uint8_t)constant;\n}\n//< Compiling Expressions make-constant\n//> Compiling Expressions emit-constant\nstatic void emitConstant(Value value) {\n emitBytes(OP_CONSTANT, makeConstant(value));\n}\n//< Compiling Expressions emit-constant\n//> Jumping Back and Forth patch-jump\nstatic void patchJump(int offset) {\n // -2 to adjust for the bytecode for the jump offset itself.\n int jump = currentChunk()->count - offset - 2;\n\n if (jump > UINT16_MAX) {\n error(\"Too much code to jump over.\");\n }\n\n currentChunk()->code[offset] = (jump >> 8) & 0xff;\n currentChunk()->code[offset + 1] = jump & 0xff;\n}\n//< Jumping Back and Forth patch-jump\n//> Local Variables init-compiler\n/* Local Variables init-compiler < Calls and Functions init-compiler\nstatic void initCompiler(Compiler* compiler) {\n*/\n//> Calls and Functions init-compiler\nstatic void initCompiler(Compiler* compiler, FunctionType type) {\n//> store-enclosing\n compiler->enclosing = current;\n//< store-enclosing\n compiler->function = NULL;\n compiler->type = type;\n//< Calls and Functions init-compiler\n compiler->localCount = 0;\n compiler->scopeDepth = 0;\n//> Calls and Functions init-function\n compiler->function = newFunction();\n//< Calls and Functions init-function\n current = compiler;\n//> Calls and Functions init-function-name\n if (type != TYPE_SCRIPT) {\n current->function->name = copyString(parser.previous.start,\n parser.previous.length);\n }\n//< Calls and Functions init-function-name\n//> Calls and Functions init-function-slot\n\n Local* local = ¤t->locals[current->localCount++];\n local->depth = 0;\n//> Closures init-zero-local-is-captured\n local->isCaptured = false;\n//< Closures init-zero-local-is-captured\n/* Calls and Functions init-function-slot < Methods and Initializers slot-zero\n local->name.start = \"\";\n local->name.length = 0;\n*/\n//> Methods and Initializers slot-zero\n if (type != TYPE_FUNCTION) {\n local->name.start = \"this\";\n local->name.length = 4;\n } else {\n local->name.start = \"\";\n local->name.length = 0;\n }\n//< Methods and Initializers slot-zero\n//< Calls and Functions init-function-slot\n}\n//< Local Variables init-compiler\n//> Compiling Expressions end-compiler\n/* Compiling Expressions end-compiler < Calls and Functions end-compiler\nstatic void endCompiler() {\n*/\n//> Calls and Functions end-compiler\nstatic ObjFunction* endCompiler() {\n//< Calls and Functions end-compiler\n emitReturn();\n//> Calls and Functions end-function\n ObjFunction* function = current->function;\n\n//< Calls and Functions end-function\n//> dump-chunk\n#ifdef DEBUG_PRINT_CODE\n if (!parser.hadError) {\n/* Compiling Expressions dump-chunk < Calls and Functions disassemble-end\n disassembleChunk(currentChunk(), \"code\");\n*/\n//> Calls and Functions disassemble-end\n disassembleChunk(currentChunk(), function->name != NULL\n ? function->name->chars : \"\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
III
\n

A Bytecode Virtual Machine

\n\n

Our Java interpreter, jlox, taught us many of the fundamentals of programming\nlanguages, but we still have much to learn. First, if you run any interesting\nLox programs in jlox, you’ll discover it’s achingly slow. The style of\ninterpretation it uses—walking the AST directly—is good enough for some\nreal-world uses, but leaves a lot to be desired for a general-purpose scripting\nlanguage.

\n

Also, we implicitly rely on runtime features of the JVM itself. We take for\ngranted that things like instanceof in Java work somehow. And we never for a\nsecond worry about memory management because the JVM’s garbage collector takes\ncare of it for us.

\n

When we were focused on high-level concepts, it was fine to gloss over those.\nBut now that we know our way around an interpreter, it’s time to dig down to\nthose lower layers and build our own virtual machine from scratch using nothing\nmore than the C standard library . . . 

\n\n\n
\n\n
\n\n\n" }, { "path": "site/a-map-of-the-territory.html", "content": "\n\n\n\nA Map of the Territory · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
2
\n

A Map of the Territory

\n\n
\n

You must have a map, no matter how rough. Otherwise you wander all over the\nplace. In The Lord of the Rings I never made anyone go farther than he could\non a given day.

\n

J. R. R. Tolkien

\n
\n

We don’t want to wander all over the place, so before we set off, let’s scan\nthe territory charted by previous language implementers. It will help us\nunderstand where we are going and the alternate routes others have taken.

\n

First, let me establish a shorthand. Much of this book is about a language’s\nimplementation, which is distinct from the language itself in some sort of\nPlatonic ideal form. Things like “stack”, “bytecode”, and “recursive descent”,\nare nuts and bolts one particular implementation might use. From the user’s\nperspective, as long as the resulting contraption faithfully follows the\nlanguage’s specification, it’s all implementation detail.

\n

We’re going to spend a lot of time on those details, so if I have to write\n“language implementation” every single time I mention them, I’ll wear my\nfingers off. Instead, I’ll use “language” to refer to either a language or an\nimplementation of it, or both, unless the distinction matters.

\n

2 . 1The Parts of a Language

\n

Engineers have been building programming languages since the Dark Ages of\ncomputing. As soon as we could talk to computers, we discovered doing so was too\nhard, and we enlisted their help. I find it fascinating that even though today’s\nmachines are literally a million times faster and have orders of magnitude more\nstorage, the way we build programming languages is virtually unchanged.

\n

Though the area explored by language designers is vast, the trails they’ve\ncarved through it are few. Not every language takes the\nexact same path—some take a shortcut or two—but otherwise they are\nreassuringly similar, from Rear Admiral Grace Hopper’s first COBOL compiler all\nthe way to some hot, new, transpile-to-JavaScript language whose “documentation”\nconsists entirely of a single, poorly edited README in a Git repository\nsomewhere.

\n\n

I visualize the network of paths an implementation may choose as climbing a\nmountain. You start off at the bottom with the program as raw source text,\nliterally just a string of characters. Each phase analyzes the program and\ntransforms it to some higher-level representation where the semantics—what\nthe author wants the computer to do—become more apparent.

\n

Eventually we reach the peak. We have a bird’s-eye view of the user’s program\nand can see what their code means. We begin our descent down the other side of\nthe mountain. We transform this highest-level representation down to\nsuccessively lower-level forms to get closer and closer to something we know how\nto make the CPU actually execute.

\"The\n

Let’s trace through each of those trails and points of interest. Our journey\nbegins on the left with the bare text of the user’s source code:

\"var\n

2 . 1 . 1Scanning

\n

The first step is scanning, also known as lexing, or (if you’re trying\nto impress someone) lexical analysis. They all mean pretty much the same\nthing. I like “lexing” because it sounds like something an evil supervillain\nwould do, but I’ll use “scanning” because it seems to be marginally more\ncommonplace.

\n

A scanner (or lexer) takes in the linear stream of characters and chunks\nthem together into a series of something more akin to “words”. In programming languages, each of these words is\ncalled a token. Some tokens are single characters, like ( and ,. Others\nmay be several characters long, like numbers (123), string literals (\"hi!\"),\nand identifiers (min).

\n\n

Some characters in a source file don’t actually mean anything. Whitespace is\noften insignificant, and comments, by definition, are ignored by the language.\nThe scanner usually discards these, leaving a clean sequence of meaningful\ntokens.

\"[var]\n

2 . 1 . 2Parsing

\n

The next step is parsing. This is where our syntax gets a grammar—the\nability to compose larger expressions and statements out of smaller parts. Did\nyou ever diagram sentences in English class? If so, you’ve done what a parser\ndoes, except that English has thousands and thousands of “keywords” and an\noverflowing cornucopia of ambiguity. Programming languages are much simpler.

\n

A parser takes the flat sequence of tokens and builds a tree structure that\nmirrors the nested nature of the grammar. These trees have a couple of different\nnames—parse tree or abstract syntax tree—depending on how\nclose to the bare syntactic structure of the source language they are. In\npractice, language hackers usually call them syntax trees, ASTs, or\noften just trees.

\"An\n

Parsing has a long, rich history in computer science that is closely tied to the\nartificial intelligence community. Many of the techniques used today to parse\nprogramming languages were originally conceived to parse human languages by AI\nresearchers who were trying to get computers to talk to us.

\n

It turns out human languages were too messy for the rigid grammars those parsers\ncould handle, but they were a perfect fit for the simpler artificial grammars of\nprogramming languages. Alas, we flawed humans still manage to use those simple\ngrammars incorrectly, so the parser’s job also includes letting us know when we\ndo by reporting syntax errors.

\n

2 . 1 . 3Static analysis

\n

The first two stages are pretty similar across all implementations. Now, the\nindividual characteristics of each language start coming into play. At this\npoint, we know the syntactic structure of the code—things like which\nexpressions are nested in which—but we don’t know much more than that.

\n

In an expression like a + b, we know we are adding a and b, but we don’t\nknow what those names refer to. Are they local variables? Global? Where are they\ndefined?

\n

The first bit of analysis that most languages do is called binding or\nresolution. For each identifier, we find out where that name is defined\nand wire the two together. This is where scope comes into play—the region\nof source code where a certain name can be used to refer to a certain\ndeclaration.

\n

If the language is statically typed, this is when we\ntype check. Once we know where a and b are declared, we can also figure out\ntheir types. Then if those types don’t support being added to each other, we\nreport a type error.

\n\n

Take a deep breath. We have attained the summit of the mountain and a sweeping\nview of the user’s program. All this semantic insight that is visible to us from\nanalysis needs to be stored somewhere. There are a few places we can squirrel it\naway:

\n
    \n
  • \n

    Often, it gets stored right back as attributes on the syntax tree\nitself—extra fields in the nodes that aren’t initialized during parsing\nbut get filled in later.

    \n
    • \n
  • \n

    Other times, we may store data in a lookup table off to the side. Typically,\nthe keys to this table are identifiers—names of variables and declarations.\nIn that case, we call it a symbol table and the values it associates with\neach key tell us what that identifier refers to.

    \n
    • \n
  • \n

    The most powerful bookkeeping tool is to transform the tree into an entirely\nnew data structure that more directly expresses the semantics of the code.\nThat’s the next section.

    \n
    • \n
\n

Everything up to this point is considered the front end of the\nimplementation. You might guess everything after this is the back end, but\nno. Back in the days of yore when “front end” and “back end” were coined,\ncompilers were much simpler. Later researchers invented new phases to stuff\nbetween the two halves. Rather than discard the old terms, William Wulf and\ncompany lumped those new phases into the charming but spatially paradoxical name\nmiddle end.

\n

2 . 1 . 4Intermediate representations

\n

You can think of the compiler as a pipeline where each stage’s job is to\norganize the data representing the user’s code in a way that makes the next\nstage simpler to implement. The front end of the pipeline is specific to the\nsource language the program is written in. The back end is concerned with the\nfinal architecture where the program will run.

\n

In the middle, the code may be stored in some intermediate\nrepresentation (IR) that isn’t tightly tied to either the source or\ndestination forms (hence “intermediate”). Instead, the IR acts as an interface\nbetween these two languages.

\n\n

This lets you support multiple source languages and target platforms with less\neffort. Say you want to implement Pascal, C, and Fortran compilers, and you want\nto target x86, ARM, and, I dunno, SPARC. Normally, that means you’re signing up\nto write nine full compilers: Pascal→x86, C→ARM, and every other\ncombination.

\n

A shared intermediate representation reduces that\ndramatically. You write one front end for each source language that produces\nthe IR. Then one back end for each target architecture. Now you can mix and\nmatch those to get every combination.

\n\n

There’s another big reason we might want to transform the code into a form that\nmakes the semantics more apparent . . . 

\n

2 . 1 . 5Optimization

\n

Once we understand what the user’s program means, we are free to swap it out\nwith a different program that has the same semantics but implements them more\nefficiently—we can optimize it.

\n

A simple example is constant folding: if some expression always evaluates to\nthe exact same value, we can do the evaluation at compile time and replace the\ncode for the expression with its result. If the user typed in this:

\n
pennyArea = 3.14159 * (0.75 / 2) * (0.75 / 2);\n
\n

we could do all of that arithmetic in the compiler and change the code to:

\n
pennyArea = 0.4417860938;\n
\n

Optimization is a huge part of the programming language business. Many language\nhackers spend their entire careers here, squeezing every drop of performance\nthey can out of their compilers to get their benchmarks a fraction of a percent\nfaster. It can become a sort of obsession.

\n

We’re mostly going to hop over that rathole in this\nbook. Many successful languages have surprisingly few compile-time\noptimizations. For example, Lua and CPython generate relatively unoptimized\ncode, and focus most of their performance effort on the runtime.

\n\n

2 . 1 . 6Code generation

\n

We have applied all of the optimizations we can think of to the user’s program.\nThe last step is converting it to a form the machine can actually run. In other\nwords, generating code (or code gen), where “code” here usually refers to\nthe kind of primitive assembly-like instructions a CPU runs and not the kind of\n“source code” a human might want to read.

\n

Finally, we are in the back end, descending the other side of the mountain.\nFrom here on out, our representation of the code becomes more and more\nprimitive, like evolution run in reverse, as we get closer to something our\nsimple-minded machine can understand.

\n

We have a decision to make. Do we generate instructions for a real CPU or a\nvirtual one? If we generate real machine code, we get an executable that the OS\ncan load directly onto the chip. Native code is lightning fast, but generating\nit is a lot of work. Today’s architectures have piles of instructions, complex\npipelines, and enough historical baggage to fill a 747’s\nluggage bay.

\n

Speaking the chip’s language also means your compiler is tied to a specific\narchitecture. If your compiler targets x86 machine code, it’s not going to\nrun on an ARM device. All the way back in the ’60s, during the\nCambrian explosion of computer architectures, that lack of portability was a\nreal obstacle.

\n\n

To get around that, hackers like Martin Richards and Niklaus Wirth, of BCPL and\nPascal fame, respectively, made their compilers produce virtual machine code.\nInstead of instructions for some real chip, they produced code for a\nhypothetical, idealized machine. Wirth called this p-code for portable,\nbut today, we generally call it bytecode because each instruction is often a\nsingle byte long.

\n

These synthetic instructions are designed to map a little more closely to the\nlanguage’s semantics, and not be so tied to the peculiarities of any one\ncomputer architecture and its accumulated historical cruft. You can think of it\nlike a dense, binary encoding of the language’s low-level operations.

\n

2 . 1 . 7Virtual machine

\n

If your compiler produces bytecode, your work isn’t over once that’s done. Since\nthere is no chip that speaks that bytecode, it’s your job to translate. Again,\nyou have two options. You can write a little mini-compiler for each target\narchitecture that converts the bytecode to native code for that machine. You\nstill have to do work for each chip you support, but\nthis last stage is pretty simple and you get to reuse the rest of the compiler\npipeline across all of the machines you support. You’re basically using your\nbytecode as an intermediate representation.

\n\n

Or you can write a virtual machine (VM), a\nprogram that emulates a hypothetical chip supporting your virtual architecture\nat runtime. Running bytecode in a VM is slower than translating it to native\ncode ahead of time because every instruction must be simulated at runtime each\ntime it executes. In return, you get simplicity and portability. Implement your\nVM in, say, C, and you can run your language on any platform that has a C\ncompiler. This is how the second interpreter we build in this book works.

\n\n

2 . 1 . 8Runtime

\n

We have finally hammered the user’s program into a form that we can execute. The\nlast step is running it. If we compiled it to machine code, we simply tell the\noperating system to load the executable and off it goes. If we compiled it to\nbytecode, we need to start up the VM and load the program into that.

\n

In both cases, for all but the basest of low-level languages, we usually need\nsome services that our language provides while the program is running. For\nexample, if the language automatically manages memory, we need a garbage\ncollector going in order to reclaim unused bits. If our language supports\n“instance of” tests so you can see what kind of object you have, then we need\nsome representation to keep track of the type of each object during execution.

\n

All of this stuff is going at runtime, so it’s called, appropriately, the\nruntime. In a fully compiled language, the code implementing the runtime\ngets inserted directly into the resulting executable. In, say, Go, each\ncompiled application has its own copy of Go’s runtime directly embedded in it.\nIf the language is run inside an interpreter or VM, then the runtime lives\nthere. This is how most implementations of languages like Java, Python, and\nJavaScript work.

\n

2 . 2Shortcuts and Alternate Routes

\n

That’s the long path covering every possible phase you might implement. Many\nlanguages do walk the entire route, but there are a few shortcuts and alternate\npaths.

\n

2 . 2 . 1Single-pass compilers

\n

Some simple compilers interleave parsing, analysis, and code generation so that\nthey produce output code directly in the parser, without ever allocating any\nsyntax trees or other IRs. These single-pass\ncompilers restrict the design of the language. You have no intermediate\ndata structures to store global information about the program, and you don’t\nrevisit any previously parsed part of the code. That means as soon as you see\nsome expression, you need to know enough to correctly compile it.

\n\n

Pascal and C were designed around this limitation. At the time, memory was so\nprecious that a compiler might not even be able to hold an entire source file\nin memory, much less the whole program. This is why Pascal’s grammar requires\ntype declarations to appear first in a block. It’s why in C you can’t call a\nfunction above the code that defines it unless you have an explicit forward\ndeclaration that tells the compiler what it needs to know to generate code for a\ncall to the later function.

\n

2 . 2 . 2Tree-walk interpreters

\n

Some programming languages begin executing code right after parsing it to an AST\n(with maybe a bit of static analysis applied). To run the program, the\ninterpreter traverses the syntax tree one branch and leaf at a time, evaluating\neach node as it goes.

\n

This implementation style is common for student projects and little languages,\nbut is not widely used for general-purpose languages\nsince it tends to be slow. Some people use “interpreter” to mean only these\nkinds of implementations, but others define that word more generally, so I’ll\nuse the inarguably explicit tree-walk interpreter to refer to these. Our\nfirst interpreter rolls this way.

\n\n

2 . 2 . 3Transpilers

\n

Writing a complete back end for a language can be a lot\nof work. If you have some existing generic IR to target, you could bolt your\nfront end onto that. Otherwise, it seems like you’re stuck. But what if you\ntreated some other source language as if it were an intermediate\nrepresentation?

\n

You write a front end for your language. Then, in the back end, instead of doing\nall the work to lower the semantics to some primitive target language, you\nproduce a string of valid source code for some other language that’s about as\nhigh level as yours. Then, you use the existing compilation tools for that\nlanguage as your escape route off the mountain and down to something you can\nexecute.

\n

They used to call this a source-to-source compiler or a transcompiler.\nAfter the rise of languages that compile to JavaScript in order to run in the\nbrowser, they’ve affected the hipster sobriquet transpiler.

\n\n

While the first transcompiler translated one assembly language to another,\ntoday, most transpilers work on higher-level languages. After the viral spread\nof UNIX to machines various and sundry, there began a long tradition of\ncompilers that produced C as their output language. C compilers were available\neverywhere UNIX was and produced efficient code, so targeting C was a good way\nto get your language running on a lot of architectures.

\n

Web browsers are the “machines” of today, and their “machine code” is\nJavaScript, so these days it seems almost every language out there has a\ncompiler that targets JS since that’s the main way to get\nyour code running in a browser.

\n\n

The front end—scanner and parser—of a transpiler looks like other\ncompilers. Then, if the source language is only a simple syntactic skin over the\ntarget language, it may skip analysis entirely and go straight to outputting the\nanalogous syntax in the destination language.

\n

If the two languages are more semantically different, you’ll see more of the\ntypical phases of a full compiler including analysis and possibly even\noptimization. Then, when it comes to code generation, instead of outputting some\nbinary language like machine code, you produce a string of grammatically correct\nsource (well, destination) code in the target language.

\n

Either way, you then run that resulting code through the output language’s\nexisting compilation pipeline, and you’re good to go.

\n

2 . 2 . 4Just-in-time compilation

\n

This last one is less a shortcut and more a dangerous alpine scramble best\nreserved for experts. The fastest way to execute code is by compiling it to\nmachine code, but you might not know what architecture your end user’s machine\nsupports. What to do?

\n

You can do the same thing that the HotSpot Java Virtual Machine (JVM),\nMicrosoft’s Common Language Runtime (CLR), and most JavaScript interpreters do.\nOn the end user’s machine, when the program is loaded—either from source in\nthe case of JS, or platform-independent bytecode for the JVM and CLR—you\ncompile it to native code for the architecture their computer supports.\nNaturally enough, this is called just-in-time compilation. Most hackers just\nsay “JIT”, pronounced like it rhymes with “fit”.

\n

The most sophisticated JITs insert profiling hooks into the generated code to\nsee which regions are most performance critical and what kind of data is flowing\nthrough them. Then, over time, they will automatically recompile those hot spots with more advanced optimizations.

\n\n

2 . 3Compilers and Interpreters

\n

Now that I’ve stuffed your head with a dictionary’s worth of programming\nlanguage jargon, we can finally address a question that’s plagued coders since\ntime immemorial: What’s the difference between a compiler and an interpreter?

\n

It turns out this is like asking the difference between a fruit and a vegetable.\nThat seems like a binary either-or choice, but actually “fruit” is a botanical\nterm and “vegetable” is culinary. One does not strictly imply the negation of\nthe other. There are fruits that aren’t vegetables (apples) and vegetables that\naren’t fruits (carrots), but also edible plants that are both fruits and\nvegetables, like tomatoes.

\n

\"A\n\n

So, back to languages:

\n
    \n
  • \n

    Compiling is an implementation technique that involves translating a\nsource language to some other—usually lower-level—form. When you\ngenerate bytecode or machine code, you are compiling. When you transpile to\nanother high-level language, you are compiling too.

    \n
    • \n
  • \n

    When we say a language implementation “is a compiler”, we mean it\ntranslates source code to some other form but doesn’t execute it. The user has\nto take the resulting output and run it themselves.

    \n
    • \n
  • \n

    Conversely, when we say an implementation “is an interpreter”, we mean it\ntakes in source code and executes it immediately. It runs programs “from\nsource”.

    \n
    • \n
\n

Like apples and oranges, some implementations are clearly compilers and not\ninterpreters. GCC and Clang take your C code and compile it to machine code. An\nend user runs that executable directly and may never even know which tool was\nused to compile it. So those are compilers for C.

\n

In older versions of Matz’s canonical implementation of Ruby, the user ran Ruby\nfrom source. The implementation parsed it and executed it directly by traversing\nthe syntax tree. No other translation occurred, either internally or in any\nuser-visible form. So this was definitely an interpreter for Ruby.

\n

But what of CPython? When you run your Python program using it, the code is\nparsed and converted to an internal bytecode format, which is then executed\ninside the VM. From the user’s perspective, this is clearly an interpreter—they run their program from source. But if you look under CPython’s scaly skin,\nyou’ll see that there is definitely some compiling going on.

\n

The answer is that it is both. CPython is an\ninterpreter, and it has a compiler. In practice, most scripting languages work\nthis way, as you can see:

\n\"A\n

That overlapping region in the center is where our second interpreter lives too,\nsince it internally compiles to bytecode. So while this book is nominally about\ninterpreters, we’ll cover some compilation too.

\n

2 . 4Our Journey

\n

That’s a lot to take in all at once. Don’t worry. This isn’t the chapter where\nyou’re expected to understand all of these pieces and parts. I just want you\nto know that they are out there and roughly how they fit together.

\n

This map should serve you well as you explore the territory beyond the guided\npath we take in this book. I want to leave you yearning to strike out on your\nown and wander all over that mountain.

\n

But, for now, it’s time for our own journey to begin. Tighten your bootlaces,\ncinch up your pack, and come along. From here on out,\nall you need to focus on is the path in front of you.

\n\n
\n

Challenges

\n
    \n
  1. \n

    Pick an open source implementation of a language you like. Download the\nsource code and poke around in it. Try to find the code that implements the\nscanner and parser. Are they handwritten, or generated using tools like\nLex and Yacc? (.l or .y files usually imply the latter.)

    \n
    2. \n
  2. \n

    Just-in-time compilation tends to be the fastest way to implement dynamically\ntyped languages, but not all of them use it. What reasons are there to not\nJIT?

    \n
    3. \n
  3. \n

    Most Lisp implementations that compile to C also contain an interpreter that\nlets them execute Lisp code on the fly as well. Why?

    \n
    4. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/a-tree-walk-interpreter.html", "content": "\n\n\n\nA Tree-Walk Interpreter · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
II
\n

A Tree-Walk Interpreter

\n\n

With this part, we begin jlox, the first of our two interpreters. Programming\nlanguages are a huge topic with piles of concepts and terminology to cram into\nyour brain all at once. Programming language theory requires a level of mental\nrigor that you probably haven’t had to summon since your last calculus final.\n(Fortunately there isn’t too much theory in this book.)

\n

Implementing an interpreter uses a few architectural tricks and design\npatterns uncommon in other kinds of applications, so we’ll be getting used to\nthe engineering side of things too. Given all of that, we’ll keep the code we\nhave to write as simple and plain as possible.

\n

In less than two thousand lines of clean Java code, we’ll build a complete\ninterpreter for Lox that implements every single feature of the language,\nexactly as we’ve specified. The first few chapters work front-to-back through\nthe phases of the interpreter—scanning, parsing, and\nevaluating code. After that, we add language features one at a time,\ngrowing a simple calculator into a full-fledged scripting language.

\n\n\n
\n\n
\n\n\n" }, { "path": "site/a-virtual-machine.html", "content": "\n\n\n\nA Virtual Machine · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
15
\n

A Virtual Machine

\n\n
\n

Magicians protect their secrets not because the secrets are large and\nimportant, but because they are so small and trivial. The wonderful effects\ncreated on stage are often the result of a secret so absurd that the magician\nwould be embarrassed to admit that that was how it was done.

\n

Christopher Priest, The Prestige

\n
\n

We’ve spent a lot of time talking about how to represent a program as a sequence\nof bytecode instructions, but it feels like learning biology using only stuffed,\ndead animals. We know what instructions are in theory, but we’ve never seen them\nin action, so it’s hard to really understand what they do. It would be hard to\nwrite a compiler that outputs bytecode when we don’t have a good understanding\nof how that bytecode behaves.

\n

So, before we go and build the front end of our new interpreter, we will begin\nwith the back end—the virtual machine that executes instructions. It breathes\nlife into the bytecode. Watching the instructions prance around gives us a\nclearer picture of how a compiler might translate the user’s source code into a\nseries of them.

\n

15 . 1An Instruction Execution Machine

\n

The virtual machine is one part of our interpreter’s internal architecture. You\nhand it a chunk of code—literally a Chunk—and it runs it. The code and\ndata structures for the VM reside in a new module.

\n
vm.h
\ncreate new file
\n
#ifndef clox_vm_h\n#define clox_vm_h\n\n#include "chunk.h"\n\ntypedef struct {\n  Chunk* chunk;\n} VM;\n\nvoid initVM();\nvoid freeVM();\n\n#endif\n
\n
vm.h, create new file
\n\n

As usual, we start simple. The VM will gradually acquire a whole pile of state\nit needs to keep track of, so we define a struct now to stuff that all in.\nCurrently, all we store is the chunk that it executes.

\n

Like we do with most of the data structures we create, we also define functions\nto create and tear down a VM. Here’s the implementation:

\n
vm.c
\ncreate new file
\n
#include "common.h"\n#include "vm.h"\n\nVM vm; \n\nvoid initVM() {\n}\n\nvoid freeVM() {\n}\n
\n
vm.c, create new file
\n\n

OK, calling those functions “implementations” is a stretch. We don’t have any\ninteresting state to initialize or free yet, so the functions are empty. Trust\nme, we’ll get there.

\n

The slightly more interesting line here is that declaration of vm. This module\nis eventually going to have a slew of functions and it would be a chore to pass\naround a pointer to the VM to all of them. Instead, we declare a single global\nVM object. We need only one anyway, and this keeps the code in the book a little\nlighter on the page.

\n\n

Before we start pumping fun code into our VM, let’s go ahead and wire it up to\nthe interpreter’s main entrypoint.

\n
int main(int argc, const char* argv[]) {\n
main.c
\nin main()
\n
  initVM();\n\n
  Chunk chunk;\n
\n
main.c, in main()
\n\n

We spin up the VM when the interpreter first starts. Then when we’re about to\nexit, we wind it down.

\n
  disassembleChunk(&chunk, "test chunk");\n
main.c
\nin main()
\n
  freeVM();\n
  freeChunk(&chunk);\n
\n
main.c, in main()
\n\n

One last ceremonial obligation:

\n
#include "debug.h"\n
main.c
\n
#include "vm.h"\n
\n\nint main(int argc, const char* argv[]) {\n
\n
main.c
\n\n

Now when you run clox, it starts up the VM before it creates that hand-authored\nchunk from the last chapter. The VM is ready and waiting, so let’s teach it\nto do something.

\n

15 . 1 . 1Executing instructions

\n

The VM springs into action when we command it to interpret a chunk of bytecode.

\n
  disassembleChunk(&chunk, "test chunk");\n
main.c
\nin main()
\n
  interpret(&chunk);\n
  freeVM();\n
\n
main.c, in main()
\n\n

This function is the main entrypoint into the VM. It’s declared like so:

\n
void freeVM();\n
vm.h
\nadd after freeVM()
\n
InterpretResult interpret(Chunk* chunk);\n
\n\n#endif\n
\n
vm.h, add after freeVM()
\n\n

The VM runs the chunk and then responds with a value from this enum:

\n
} VM;\n\n
vm.h
\nadd after struct VM
\n
typedef enum {\n  INTERPRET_OK,\n  INTERPRET_COMPILE_ERROR,\n  INTERPRET_RUNTIME_ERROR\n} InterpretResult;\n\n
void initVM();\nvoid freeVM();\n
\n
vm.h, add after struct VM
\n\n

We aren’t using the result yet, but when we have a compiler that reports static\nerrors and a VM that detects runtime errors, the interpreter will use this to\nknow how to set the exit code of the process.

\n

We’re inching towards some actual implementation.

\n
vm.c
\nadd after freeVM()
\n
InterpretResult interpret(Chunk* chunk) {\n  vm.chunk = chunk;\n  vm.ip = vm.chunk->code;\n  return run();\n}\n
\n
vm.c, add after freeVM()
\n\n

First, we store the chunk being executed in the VM. Then we call run(), an\ninternal helper function that actually runs the bytecode instructions. Between\nthose two parts is an intriguing line. What is this ip business?

\n

As the VM works its way through the bytecode, it keeps track of where it is—the location of the instruction currently being executed. We don’t use a local variable inside run() for this because eventually\nother functions will need to access it. Instead, we store it as a field in VM.

\n\n
typedef struct {\n  Chunk* chunk;\n
vm.h
\nin struct VM
\n
  uint8_t* ip;\n
} VM;\n
\n
vm.h, in struct VM
\n\n

Its type is a byte pointer. We use an actual real C pointer pointing right into\nthe middle of the bytecode array instead of something like an integer index\nbecause it’s faster to dereference a pointer than look up an element in an array\nby index.

\n

The name “IP” is traditional, and—unlike many traditional names in CS—actually makes sense: it’s an instruction pointer. Almost every\ninstruction set in the world, real and virtual, has a\nregister or variable like this.

\n\n

We initialize ip by pointing it at the first byte of code in the chunk. We\nhaven’t executed that instruction yet, so ip points to the instruction about\nto be executed. This will be true during the entire time the VM is running: the\nIP always points to the next instruction, not the one currently being handled.

\n

The real fun happens in run().

\n
vm.c
\nadd after freeVM()
\n
static InterpretResult run() {\n#define READ_BYTE() (*vm.ip++)\n\n  for (;;) {\n    uint8_t instruction;\n    switch (instruction = READ_BYTE()) {\n      case OP_RETURN: {\n        return INTERPRET_OK;\n      }\n    }\n  }\n\n#undef READ_BYTE\n}\n
\n
vm.c, add after freeVM()
\n\n

This is the single most important function in all\nof clox, by far. When the interpreter executes a user’s program, it will spend\nsomething like 90% of its time inside run(). It is the beating heart of the\nVM.

\n\n

Despite that dramatic intro, it’s conceptually pretty simple. We have an outer\nloop that goes and goes. Each turn through that loop, we read and execute a\nsingle bytecode instruction.

\n

To process an instruction, we first figure out what kind of instruction we’re\ndealing with. The READ_BYTE macro reads the byte currently pointed at by ip\nand then advances the instruction pointer. The first\nbyte of any instruction is the opcode. Given a numeric opcode, we need to get to\nthe right C code that implements that instruction’s semantics. This process is\ncalled decoding or dispatching the instruction.

\n\n

We do that process for every single instruction, every single time one is\nexecuted, so this is the most performance critical part of the entire virtual\nmachine. Programming language lore is filled with clever techniques to do bytecode dispatch efficiently,\ngoing all the way back to the early days of computers.

\n\n

Alas, the fastest solutions require either non-standard extensions to C, or\nhandwritten assembly code. For clox, we’ll keep it simple. Just like our\ndisassembler, we have a single giant switch statement with a case for each\nopcode. The body of each case implements that opcode’s behavior.

\n

So far, we handle only a single instruction, OP_RETURN, and the only thing it\ndoes is exit the loop entirely. Eventually, that instruction will be used to\nreturn from the current Lox function, but we don’t have functions yet, so we’ll\nrepurpose it temporarily to end the execution.

\n

Let’s go ahead and support our one other instruction.

\n
    switch (instruction = READ_BYTE()) {\n
vm.c
\nin run()
\n
      case OP_CONSTANT: {\n        Value constant = READ_CONSTANT();\n        printValue(constant);\n        printf("\\n");\n        break;\n      }\n
      case OP_RETURN: {\n
\n
vm.c, in run()
\n\n

We don’t have enough machinery in place yet to do anything useful with a\nconstant. For now, we’ll just print it out so we interpreter hackers can see\nwhat’s going on inside our VM. That call to printf() necessitates an include.

\n
vm.c
\nadd to top of file
\n
#include <stdio.h>\n\n
#include "common.h"\n
\n
vm.c, add to top of file
\n\n

We also have a new macro to define.

\n
#define READ_BYTE() (*vm.ip++)\n
vm.c
\nin run()
\n
#define READ_CONSTANT() (vm.chunk->constants.values[READ_BYTE()])\n
\n\n  for (;;) {\n
\n
vm.c, in run()
\n\n

READ_CONSTANT() reads the next byte from the bytecode, treats the resulting\nnumber as an index, and looks up the corresponding Value in the chunk’s constant\ntable. In later chapters, we’ll add a few more instructions with operands that\nrefer to constants, so we’re setting up this helper macro now.

\n

Like the previous READ_BYTE macro, READ_CONSTANT is only used inside\nrun(). To make that scoping more explicit, the macro definitions themselves\nare confined to that function. We define them at the\nbeginning and—because we care—undefine them at the end.

\n
#undef READ_BYTE\n
vm.c
\nin run()
\n
#undef READ_CONSTANT\n
}\n
\n
vm.c, in run()
\n\n\n

15 . 1 . 2Execution tracing

\n

If you run clox now, it executes the chunk we hand-authored in the last chapter\nand spits out 1.2 to your terminal. We can see that it’s working, but that’s\nonly because our implementation of OP_CONSTANT has temporary code to log the\nvalue. Once that instruction is doing what it’s supposed to do and plumbing that\nconstant along to other operations that want to consume it, the VM will become a\nblack box. That makes our lives as VM implementers harder.

\n

To help ourselves out, now is a good time to add some diagnostic logging to the\nVM like we did with chunks themselves. In fact, we’ll even reuse the same code.\nWe don’t want this logging enabled all the time—it’s just for us VM hackers,\nnot Lox users—so first we create a flag to hide it behind.

\n
#include <stdint.h>\n
common.h
\n
\n\n#define DEBUG_TRACE_EXECUTION\n
\n\n#endif\n
\n
common.h
\n\n

When this flag is defined, the VM disassembles and prints each instruction right\nbefore executing it. Where our previous disassembler walked an entire chunk\nonce, statically, this disassembles instructions dynamically, on the fly.

\n
  for (;;) {\n
vm.c
\nin run()
\n
#ifdef DEBUG_TRACE_EXECUTION\n    disassembleInstruction(vm.chunk,\n                           (int)(vm.ip - vm.chunk->code));\n#endif\n\n
    uint8_t instruction;\n
\n
vm.c, in run()
\n\n

Since disassembleInstruction() takes an integer byte offset and we store the\ncurrent instruction reference as a direct pointer, we first do a little pointer\nmath to convert ip back to a relative offset from the beginning of the\nbytecode. Then we disassemble the instruction that begins at that byte.

\n

As ever, we need to bring in the declaration of the function before we can call\nit.

\n
#include "common.h"\n
vm.c
\n
#include "debug.h"\n
#include "vm.h"\n
\n
vm.c
\n\n

I know this code isn’t super impressive so far—it’s literally a switch\nstatement wrapped in a for loop but, believe it or not, this is one of the two\nmajor components of our VM. With this, we can imperatively execute instructions.\nIts simplicity is a virtue—the less work it does, the faster it can do it.\nContrast this with all of the complexity and overhead we had in jlox with the\nVisitor pattern for walking the AST.

\n

15 . 2A Value Stack Manipulator

\n

In addition to imperative side effects, Lox has expressions that produce,\nmodify, and consume values. Thus, our compiled bytecode needs a way to shuttle\nvalues around between the different instructions that need them. For example:

\n
print 3 - 2;\n
\n

We obviously need instructions for the constants 3 and 2, the print statement,\nand the subtraction. But how does the subtraction instruction know that 3 is\nthe minuend and 2 is the subtrahend? How does the print\ninstruction know to print the result of that?

\n\n

To put a finer point on it, look at this thing right here:

\n
fun echo(n) {\n  print n;\n  return n;\n}\n\nprint echo(echo(1) + echo(2)) + echo(echo(4) + echo(5));\n
\n

I wrapped each subexpression in a call to echo() that prints and returns its\nargument. That side effect means we can see the exact order of operations.

\n

Don’t worry about the VM for a minute. Think about just the semantics of Lox\nitself. The operands to an arithmetic operator obviously need to be evaluated\nbefore we can perform the operation itself. (It’s pretty hard to add a + b if\nyou don’t know what a and b are.) Also, when we implemented expressions in\njlox, we decided that the left operand must be\nevaluated before the right.

\n\n

Here is the syntax tree for the print statement:

\n

\"The

\n

Given left-to-right evaluation, and the way the expressions are nested, any\ncorrect Lox implementation must print these numbers in this order:

\n
1  // from echo(1)\n2  // from echo(2)\n3  // from echo(1 + 2)\n4  // from echo(4)\n5  // from echo(5)\n9  // from echo(4 + 5)\n12 // from print 3 + 9\n
\n

Our old jlox interpreter accomplishes this by recursively traversing the AST. It\ndoes a postorder traversal. First it recurses down the left operand branch,\nthen the right operand, then finally it evaluates the node itself.

\n

After evaluating the left operand, jlox needs to store that result somewhere\ntemporarily while it’s busy traversing down through the right operand tree. We\nuse a local variable in Java for that. Our recursive tree-walk interpreter\ncreates a unique Java call frame for each node being evaluated, so we could have\nas many of these local variables as we needed.

\n

In clox, our run() function is not recursive—the nested expression tree is\nflattened out into a linear series of instructions. We don’t have the luxury of\nusing C local variables, so how and where should we store these temporary\nvalues? You can probably guess already, but I want to\nreally drill into this because it’s an aspect of programming that we take for\ngranted, but we rarely learn why computers are architected this way.

\n\n

Let’s do a weird exercise. We’ll walk through the execution of the above program\na step at a time:

\n

\"The

\n

On the left are the steps of code. On the right are the values we’re tracking.\nEach bar represents a number. It starts when the value is first produced—either a constant or the result of an addition. The length of the bar tracks\nwhen a previously produced value needs to be kept around, and it ends when that\nvalue finally gets consumed by an operation.

\n

As you step through, you see values appear and then later get eaten. The\nlongest-lived ones are the values produced from the left-hand side of an\naddition. Those stick around while we work through the right-hand operand\nexpression.

\n

In the above diagram, I gave each unique number its own visual column. Let’s be\na little more parsimonious. Once a number is consumed, we allow its column to be\nreused for another later value. In other words, we take all of those gaps\nup there and fill them in, pushing in numbers from the right:

\n

\"Like

\n

There’s some interesting stuff going on here. When we shift everything over,\neach number still manages to stay in a single column for its entire life. Also,\nthere are no gaps left. In other words, whenever a number appears earlier than\nanother, then it will live at least as long as that second one. The first number\nto appear is the last to be consumed. Hmm . . . last-in, first-out . . . why, that’s a\nstack!

\n\n

In the second diagram, each time we introduce a number, we push it onto the\nstack from the right. When numbers are consumed, they are always popped off from\nrightmost to left.

\n

Since the temporary values we need to track naturally have stack-like behavior,\nour VM will use a stack to manage them. When an instruction “produces” a value,\nit pushes it onto the stack. When it needs to consume one or more values, it\ngets them by popping them off the stack.

\n

15 . 2 . 1The VM’s Stack

\n

Maybe this doesn’t seem like a revelation, but I love stack-based VMs. When\nyou first see a magic trick, it feels like something actually magical. But then\nyou learn how it works—usually some mechanical gimmick or misdirection—and\nthe sense of wonder evaporates. There are a couple of\nideas in computer science where even after I pulled them apart and learned all\nthe ins and outs, some of the initial sparkle remained. Stack-based VMs are one\nof those.

\n\n

As you’ll see in this chapter, executing instructions in a stack-based VM is\ndead simple. In later chapters, you’ll also discover\nthat compiling a source language to a stack-based instruction set is a piece of\ncake. And yet, this architecture is fast enough to be used by production\nlanguage implementations. It almost feels like cheating at the programming\nlanguage game.

\n\n

Alrighty, it’s codin’ time! Here’s the stack:

\n
typedef struct {\n  Chunk* chunk;\n  uint8_t* ip;\n
vm.h
\nin struct VM
\n
  Value stack[STACK_MAX];\n  Value* stackTop;\n
} VM;\n
\n
vm.h, in struct VM
\n\n

We implement the stack semantics ourselves on top of a raw C array. The bottom\nof the stack—the first value pushed and the last to be popped—is at\nelement zero in the array, and later pushed values follow it. If we push the\nletters of “crepe”—my favorite stackable breakfast item—onto the stack, in\norder, the resulting C array looks like this:

\n

\"An

\n

Since the stack grows and shrinks as values are pushed and popped, we need to\ntrack where the top of the stack is in the array. As with ip, we use a direct\npointer instead of an integer index since it’s faster to dereference the pointer\nthan calculate the offset from the index each time we need it.

\n

The pointer points at the array element just past the element containing the\ntop value on the stack. That seems a little odd, but almost every implementation\ndoes this. It means we can indicate that the stack is empty by pointing at\nelement zero in the array.

\n

\"An

\n

If we pointed to the top element, then for an empty stack we’d need to point at\nelement -1. That’s undefined in C. As we push values\nonto the stack . . . 

\n\n

\"An

\n

 . . . stackTop always points just past the last item.

\n

\"An

\n

I remember it like this: stackTop points to where the next value to be pushed\nwill go. The maximum number of values we can store on the stack (for now, at\nleast) is:

\n
#include "chunk.h"\n
vm.h
\n
\n\n#define STACK_MAX 256\n
\n\ntypedef struct {\n
\n
vm.h
\n\n

Giving our VM a fixed stack size means it’s possible for some sequence of\ninstructions to push too many values and run out of stack space—the classic\n“stack overflow”. We could grow the stack dynamically as needed, but for now\nwe’ll keep it simple. Since VM uses Value, we need to include its declaration.

\n
#include "chunk.h"\n
vm.h
\n
#include "value.h"\n
\n\n#define STACK_MAX 256\n
\n
vm.h
\n\n

Now that VM has some interesting state, we get to initialize it.

\n
void initVM() {\n
vm.c
\nin initVM()
\n
  resetStack();\n
}\n
\n
vm.c, in initVM()
\n\n

That uses this helper function:

\n
vm.c
\nadd after variable vm
\n
static void resetStack() {\n  vm.stackTop = vm.stack;\n}\n
\n
vm.c, add after variable vm
\n\n

Since the stack array is declared directly inline in the VM struct, we don’t\nneed to allocate it. We don’t even need to clear the unused cells in the\narray—we simply won’t access them until after values have been stored in\nthem. The only initialization we need is to set stackTop to point to the\nbeginning of the array to indicate that the stack is empty.

\n

The stack protocol supports two operations:

\n
InterpretResult interpret(Chunk* chunk);\n
vm.h
\nadd after interpret()
\n
void push(Value value);\nValue pop();\n
\n\n#endif\n
\n
vm.h, add after interpret()
\n\n

You can push a new value onto the top of the stack, and you can pop the most\nrecently pushed value back off. Here’s the first function:

\n
vm.c
\nadd after freeVM()
\n
void push(Value value) {\n  *vm.stackTop = value;\n  vm.stackTop++;\n}\n
\n
vm.c, add after freeVM()
\n\n

If you’re rusty on your C pointer syntax and operations, this is a good warm-up.\nThe first line stores value in the array element at the top of the stack.\nRemember, stackTop points just past the last used element, at the next\navailable one. This stores the value in that slot. Then we increment the pointer\nitself to point to the next unused slot in the array now that the previous slot\nis occupied.

\n

Popping is the mirror image.

\n
vm.c
\nadd after push()
\n
Value pop() {\n  vm.stackTop--;\n  return *vm.stackTop;\n}\n
\n
vm.c, add after push()
\n\n

First, we move the stack pointer back to get to the most recent used slot in\nthe array. Then we look up the value at that index and return it. We don’t need\nto explicitly “remove” it from the array—moving stackTop down is enough to\nmark that slot as no longer in use.

\n

15 . 2 . 2Stack tracing

\n

We have a working stack, but it’s hard to see that it’s working. When we start\nimplementing more complex instructions and compiling and running larger pieces\nof code, we’ll end up with a lot of values crammed into that array. It would\nmake our lives as VM hackers easier if we had some visibility into the stack.

\n

To that end, whenever we’re tracing execution, we’ll also show the current\ncontents of the stack before we interpret each instruction.

\n
#ifdef DEBUG_TRACE_EXECUTION\n
vm.c
\nin run()
\n
    printf("          ");\n    for (Value* slot = vm.stack; slot < vm.stackTop; slot++) {\n      printf("[ ");\n      printValue(*slot);\n      printf(" ]");\n    }\n    printf("\\n");\n
    disassembleInstruction(vm.chunk,\n
\n
vm.c, in run()
\n\n

We loop, printing each value in the array, starting at the first (bottom of the\nstack) and ending when we reach the top. This lets us observe the effect of each\ninstruction on the stack. The output is pretty verbose, but it’s useful when\nwe’re surgically extracting a nasty bug from the bowels of the interpreter.

\n

Stack in hand, let’s revisit our two instructions. First up:

\n
      case OP_CONSTANT: {\n        Value constant = READ_CONSTANT();\n
vm.c
\nin run()
\nreplace 2 lines
\n
        push(constant);\n
        break;\n
\n
vm.c, in run(), replace 2 lines
\n\n

In the last chapter, I was hand-wavey about how the OP_CONSTANT instruction\n“loads” a constant. Now that we have a stack you know what it means to actually\nproduce a value: it gets pushed onto the stack.

\n
      case OP_RETURN: {\n
vm.c
\nin run()
\n
        printValue(pop());\n        printf("\\n");\n
        return INTERPRET_OK;\n
\n
vm.c, in run()
\n\n

Then we make OP_RETURN pop the stack and print the top value before exiting.\nWhen we add support for real functions to clox, we’ll change this code. But, for\nnow, it gives us a way to get the VM executing simple instruction sequences and\ndisplaying the result.

\n

15 . 3An Arithmetic Calculator

\n

The heart and soul of our VM are in place now. The bytecode loop dispatches and\nexecutes instructions. The stack grows and shrinks as values flow through it.\nThe two halves work, but it’s hard to get a feel for how cleverly they interact\nwith only the two rudimentary instructions we have so far. So let’s teach our\ninterpreter to do arithmetic.

\n

We’ll start with the simplest arithmetic operation, unary negation.

\n
var a = 1.2;\nprint -a; // -1.2.\n
\n

The prefix - operator takes one operand, the value to negate. It produces a\nsingle result. We aren’t fussing with a parser yet, but we can add the\nbytecode instruction that the above syntax will compile to.

\n
  OP_CONSTANT,\n
chunk.h
\nin enum OpCode
\n
  OP_NEGATE,\n
  OP_RETURN,\n
\n
chunk.h, in enum OpCode
\n\n

We execute it like so:

\n
      }\n
vm.c
\nin run()
\n
      case OP_NEGATE:   push(-pop()); break;\n
      case OP_RETURN: {\n
\n
vm.c, in run()
\n\n

The instruction needs a value to operate on, which it gets by popping from the\nstack. It negates that, then pushes the result back on for later instructions to\nuse. Doesn’t get much easier than that. We can disassemble it too.

\n
    case OP_CONSTANT:\n      return constantInstruction("OP_CONSTANT", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_NEGATE:\n      return simpleInstruction("OP_NEGATE", offset);\n
    case OP_RETURN:\n
\n
debug.c, in disassembleInstruction()
\n\n

And we can try it out in our test chunk.

\n
  writeChunk(&chunk, constant, 123);\n
main.c
\nin main()
\n
  writeChunk(&chunk, OP_NEGATE, 123);\n
\n\n  writeChunk(&chunk, OP_RETURN, 123);\n
\n
main.c, in main()
\n\n

After loading the constant, but before returning, we execute the negate\ninstruction. That replaces the constant on the stack with its negation. Then the\nreturn instruction prints that out:

\n
-1.2\n
\n

Magical!

\n

15 . 3 . 1Binary operators

\n

OK, unary operators aren’t that impressive. We still only ever have a single\nvalue on the stack. To really see some depth, we need binary operators. Lox has\nfour binary arithmetic operators: addition, subtraction,\nmultiplication, and division. We’ll go ahead and implement them all at the same\ntime.

\n\n
  OP_CONSTANT,\n
chunk.h
\nin enum OpCode
\n
  OP_ADD,\n  OP_SUBTRACT,\n  OP_MULTIPLY,\n  OP_DIVIDE,\n
  OP_NEGATE,\n
\n
chunk.h, in enum OpCode
\n\n

Back in the bytecode loop, they are executed like this:

\n
      }\n
vm.c
\nin run()
\n
      case OP_ADD:      BINARY_OP(+); break;\n      case OP_SUBTRACT: BINARY_OP(-); break;\n      case OP_MULTIPLY: BINARY_OP(*); break;\n      case OP_DIVIDE:   BINARY_OP(/); break;\n
      case OP_NEGATE:   push(-pop()); break;\n
\n
vm.c, in run()
\n\n

The only difference between these four instructions is which underlying C\noperator they ultimately use to combine the two operands. Surrounding that core\narithmetic expression is some boilerplate code to pull values off the stack and\npush the result. When we later add dynamic typing, that boilerplate will grow.\nTo avoid repeating that code four times, I wrapped it up in a macro.

\n
#define READ_CONSTANT() (vm.chunk->constants.values[READ_BYTE()])\n
vm.c
\nin run()
\n
#define BINARY_OP(op) \\\n    do { \\\n      double b = pop(); \\\n      double a = pop(); \\\n      push(a op b); \\\n    } while (false)\n
\n\n  for (;;) {\n
\n
vm.c, in run()
\n\n

I admit this is a fairly adventurous use of the C\npreprocessor. I hesitated to do this, but you’ll be glad in later chapters when\nwe need to add the type checking for each operand and stuff. It would be a chore\nto walk you through the same code four times.

\n\n

If you aren’t familiar with the trick already, that outer do while loop\nprobably looks really weird. This macro needs to expand to a series of\nstatements. To be careful macro authors, we want to ensure those statements all\nend up in the same scope when the macro is expanded. Imagine if you defined:

\n
#define WAKE_UP() makeCoffee(); drinkCoffee();\n
\n

And then used it like:

\n
if (morning) WAKE_UP();\n
\n

The intent is to execute both statements of the macro body only if morning is\ntrue. But it expands to:

\n
if (morning) makeCoffee(); drinkCoffee();;\n
\n

Oops. The if attaches only to the first statement. You might think you could\nfix this using a block.

\n
#define WAKE_UP() { makeCoffee(); drinkCoffee(); }\n
\n

That’s better, but you still risk:

\n
if (morning)\n  WAKE_UP();\nelse\n  sleepIn();\n
\n

Now you get a compile error on the else because of that trailing ; after the\nmacro’s block. Using a do while loop in the macro looks funny, but it gives\nyou a way to contain multiple statements inside a block that also permits a\nsemicolon at the end.

\n

Where were we? Right, so what the body of that macro does is straightforward. A\nbinary operator takes two operands, so it pops twice. It performs the operation\non those two values and then pushes the result.

\n

Pay close attention to the order of the two pops. Note that we assign the\nfirst popped operand to b, not a. It looks backwards. When the operands\nthemselves are calculated, the left is evaluated first, then the right. That\nmeans the left operand gets pushed before the right operand. So the right\noperand will be on top of the stack. Thus, the first value we pop is b.

\n

For example, if we compile 3 - 1, the data flow between the instructions looks\nlike so:

\n

\"A

\n

As we did with the other macros inside run(), we clean up after ourselves at\nthe end of the function.

\n
#undef READ_CONSTANT\n
vm.c
\nin run()
\n
#undef BINARY_OP\n
}\n
\n
vm.c, in run()
\n\n

Last is disassembler support.

\n
    case OP_CONSTANT:\n      return constantInstruction("OP_CONSTANT", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_ADD:\n      return simpleInstruction("OP_ADD", offset);\n    case OP_SUBTRACT:\n      return simpleInstruction("OP_SUBTRACT", offset);\n    case OP_MULTIPLY:\n      return simpleInstruction("OP_MULTIPLY", offset);\n    case OP_DIVIDE:\n      return simpleInstruction("OP_DIVIDE", offset);\n
    case OP_NEGATE:\n
\n
debug.c, in disassembleInstruction()
\n\n

The arithmetic instruction formats are simple, like OP_RETURN. Even though the\narithmetic operators take operands—which are found on the stack—the\narithmetic bytecode instructions do not.

\n

Let’s put some of our new instructions through their paces by evaluating a\nlarger expression:

\n

\"The

\n

Building on our existing example chunk, here’s the additional instructions we\nneed to hand-compile that AST to bytecode.

\n
  int constant = addConstant(&chunk, 1.2);\n  writeChunk(&chunk, OP_CONSTANT, 123);\n  writeChunk(&chunk, constant, 123);\n
main.c
\nin main()
\n
\n\n  constant = addConstant(&chunk, 3.4);\n  writeChunk(&chunk, OP_CONSTANT, 123);\n  writeChunk(&chunk, constant, 123);\n\n  writeChunk(&chunk, OP_ADD, 123);\n\n  constant = addConstant(&chunk, 5.6);\n  writeChunk(&chunk, OP_CONSTANT, 123);\n  writeChunk(&chunk, constant, 123);\n\n  writeChunk(&chunk, OP_DIVIDE, 123);\n
  writeChunk(&chunk, OP_NEGATE, 123);\n\n  writeChunk(&chunk, OP_RETURN, 123);\n
\n
main.c, in main()
\n\n

The addition goes first. The instruction for the left constant, 1.2, is already\nthere, so we add another for 3.4. Then we add those two using OP_ADD, leaving\nit on the stack. That covers the left side of the division. Next we push the\n5.6, and divide the result of the addition by it. Finally, we negate the result\nof that.

\n

Note how the output of the OP_ADD implicitly flows into being an operand of\nOP_DIVIDE without either instruction being directly coupled to each other.\nThat’s the magic of the stack. It lets us freely compose instructions without\nthem needing any complexity or awareness of the data flow. The stack acts like a\nshared workspace that they all read from and write to.

\n

In this tiny example chunk, the stack still only gets two values tall, but when\nwe start compiling Lox source to bytecode, we’ll have chunks that use much more\nof the stack. In the meantime, try playing around with this hand-authored chunk\nto calculate different nested arithmetic expressions and see how values flow\nthrough the instructions and stack.

\n

You may as well get it out of your system now. This is the last chunk we’ll\nbuild by hand. When we next revisit bytecode, we will be writing a compiler to\ngenerate it for us.

\n
\n

Challenges

\n
    \n
  1. \n

    What bytecode instruction sequences would you generate for the following\nexpressions:

    \n
    1 * 2 + 3\n1 + 2 * 3\n3 - 2 - 1\n1 + 2 * 3 - 4 / -5\n
    \n

    (Remember that Lox does not have a syntax for negative number literals, so\nthe -5 is negating the number 5.)

    \n
    2. \n
  2. \n

    If we really wanted a minimal instruction set, we could eliminate either\nOP_NEGATE or OP_SUBTRACT. Show the bytecode instruction sequence you\nwould generate for:

    \n
    4 - 3 * -2\n
    \n

    First, without using OP_NEGATE. Then, without using OP_SUBTRACT.

    \n

    Given the above, do you think it makes sense to have both instructions? Why\nor why not? Are there any other redundant instructions you would consider\nincluding?

    \n
    3. \n
  3. \n

    Our VM’s stack has a fixed size, and we don’t check if pushing a value\noverflows it. This means the wrong series of instructions could cause our\ninterpreter to crash or go into undefined behavior. Avoid that by\ndynamically growing the stack as needed.

    \n

    What are the costs and benefits of doing so?

    \n
    4. \n
  4. \n

    To interpret OP_NEGATE, we pop the operand, negate the value, and then\npush the result. That’s a simple implementation, but it increments and\ndecrements stackTop unnecessarily, since the stack ends up the same height\nin the end. It might be faster to simply negate the value in place on the\nstack and leave stackTop alone. Try that and see if you can measure a\nperformance difference.

    \n

    Are there other instructions where you can do a similar optimization?

    \n
    5. \n
\n
\n
\n

Design Note: Register-Based Bytecode

\n

For the remainder of this book, we’ll meticulously implement an interpreter\naround a stack-based bytecode instruction set. There’s another family of\nbytecode architectures out there—register-based. Despite the name, these\nbytecode instructions aren’t quite as difficult to work with as the registers in\nan actual chip like x64. With real hardware registers,\nyou usually have only a handful for the entire program, so you spend a lot of\neffort trying to use them efficiently and shuttling stuff in and out of\nthem.

\n\n

In a register-based VM, you still have a stack. Temporary values still get\npushed onto it and popped when no longer needed. The main difference is that\ninstructions can read their inputs from anywhere in the stack and can store\ntheir outputs into specific stack slots.

\n

Take this little Lox script:

\n
var a = 1;\nvar b = 2;\nvar c = a + b;\n
\n

In our stack-based VM, the last statement will get compiled to something like:

\n
load <a>  // Read local variable a and push onto stack.\nload <b>  // Read local variable b and push onto stack.\nadd       // Pop two values, add, push result.\nstore <c> // Pop value and store in local variable c.\n
\n

(Don’t worry if you don’t fully understand the load and store instructions yet.\nWe’ll go over them in much greater detail when we implement\nvariables.) We have four separate instructions. That means four\ntimes through the bytecode interpret loop, four instructions to decode and\ndispatch. It’s at least seven bytes of code—four for the opcodes and another\nthree for the operands identifying which locals to load and store. Three pushes\nand three pops. A lot of work!

\n

In a register-based instruction set, instructions can read from and store\ndirectly into local variables. The bytecode for the last statement above looks\nlike:

\n
add <a> <b> <c> // Read values from a and b, add, store in c.\n
\n

The add instruction is bigger—it has three instruction operands that define\nwhere in the stack it reads its inputs from and writes the result to. But since\nlocal variables live on the stack, it can read directly from a and b and\nthen store the result right into c.

\n

There’s only a single instruction to decode and dispatch, and the whole thing\nfits in four bytes. Decoding is more complex because of the additional operands,\nbut it’s still a net win. There’s no pushing and popping or other stack\nmanipulation.

\n

The main implementation of Lua used to be stack-based. For Lua\n5.0, the implementers switched to a register instruction set and noted a\nspeed improvement. The amount of improvement, naturally, depends heavily on the\ndetails of the language semantics, specific instruction set, and compiler\nsophistication, but that should get your attention.

\n\n

That raises the obvious question of why I’m going to spend the rest of the book\ndoing a stack-based bytecode. Register VMs are neat, but they are quite a bit\nharder to write a compiler for. For what is likely to be your very first\ncompiler, I wanted to stick with an instruction set that’s easy to generate and\neasy to execute. Stack-based bytecode is marvelously simple.

\n

It’s also much better known in the literature and the community. Even though\nyou may eventually move to something more advanced, it’s a good common ground to\nshare with the rest of your language hacker peers.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/acknowledgements.html", "content": "\n\n\n\nAcknowledgements · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n

Acknowledgements

\n\n

When the first copy of “Game Programming Patterns” sold, I guess I had\nthe right to call myself an author. But it took time to feel comfortable with\nthat label. Thank you to everyone who bought copies of my first book, and to the\npublishers and translators who brought it to other languages. You gave me the\nconfidence to believe I could tackle a project of this scope. Well, that, and\nmassively underestimating what I was getting myself into, but that’s on me.

\n

A fear particular to technical writing is getting stuff wrong. Tests and\nstatic analysis only get you so far. Once the code and prose is in ink on paper,\nthere’s no fixing it. I am deeply grateful to the many people who filed issues\nand pull requests on the open source repo for the book. Special thanks\ngo to cm1776, who filed 145 tactfully worded issues pointing out hundreds of\ncode errors, typos, and unclear sentences. The book is more accurate and\nreadable because of you all.

\n

I’m grateful to my copy editor Kari Somerton who braved a heap of computer\nscience jargon and an unfamilar workflow in order to fix my many grammar errors\nand stylistic inconsistencies.

\n

When the pandemic turned everyone’s life upside down, a number of people reached\nout to tell me that my book provided a helpful distraction. This book that I\nspent six years writing forms a chapter in my own life’s story and I’m grateful\nto the readers who contacted me and made that chapter more meaningful.

\n

Finally, the deepest thanks go to my wife Megan and my daughters Lily and\nGretchen. You patiently endured the time I had to sink into the book, and my\nstress while writing it. There’s no one I’d rather be stuck at home with.

\n\n\n
\n\n
\n\n\n" }, { "path": "site/appendix-i.html", "content": "\n\n\n\nAppendix I · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
A1
\n

Appendix I

\n\n

Here is a complete grammar for Lox. The chapters that introduce each part of the\nlanguage include the grammar rules there, but this collects them all into one\nplace.

\n

A1 . 1Syntax Grammar

\n

The syntactic grammar is used to parse the linear sequence of tokens into the\nnested syntax tree structure. It starts with the first rule that matches an\nentire Lox program (or a single REPL entry).

\n
program        → declaration* EOF ;\n
\n

A1 . 1 . 1Declarations

\n

A program is a series of declarations, which are the statements that bind new\nidentifiers or any of the other statement types.

\n
declaration    → classDecl\n               | funDecl\n               | varDecl\n               | statement ;\n\nclassDecl      → "class" IDENTIFIER ( "<" IDENTIFIER )?\n                 "{" function* "}" ;\nfunDecl        → "fun" function ;\nvarDecl        → "var" IDENTIFIER ( "=" expression )? ";" ;\n
\n

A1 . 1 . 2Statements

\n

The remaining statement rules produce side effects, but do not introduce\nbindings.

\n
statement      → exprStmt\n               | forStmt\n               | ifStmt\n               | printStmt\n               | returnStmt\n               | whileStmt\n               | block ;\n\nexprStmt       → expression ";" ;\nforStmt        → "for" "(" ( varDecl | exprStmt | ";" )\n                           expression? ";"\n                           expression? ")" statement ;\nifStmt         → "if" "(" expression ")" statement\n                 ( "else" statement )? ;\nprintStmt      → "print" expression ";" ;\nreturnStmt     → "return" expression? ";" ;\nwhileStmt      → "while" "(" expression ")" statement ;\nblock          → "{" declaration* "}" ;\n
\n

Note that block is a statement rule, but is also used as a nonterminal in a\ncouple of other rules for things like function bodies.

\n

A1 . 1 . 3Expressions

\n

Expressions produce values. Lox has a number of unary and binary operators with\ndifferent levels of precedence. Some grammars for languages do not directly\nencode the precedence relationships and specify that elsewhere. Here, we use a\nseparate rule for each precedence level to make it explicit.

\n
expression     → assignment ;\n\nassignment     → ( call "." )? IDENTIFIER "=" assignment\n               | logic_or ;\n\nlogic_or       → logic_and ( "or" logic_and )* ;\nlogic_and      → equality ( "and" equality )* ;\nequality       → comparison ( ( "!=" | "==" ) comparison )* ;\ncomparison     → term ( ( ">" | ">=" | "<" | "<=" ) term )* ;\nterm           → factor ( ( "-" | "+" ) factor )* ;\nfactor         → unary ( ( "/" | "*" ) unary )* ;\n\nunary          → ( "!" | "-" ) unary | call ;\ncall           → primary ( "(" arguments? ")" | "." IDENTIFIER )* ;\nprimary        → "true" | "false" | "nil" | "this"\n               | NUMBER | STRING | IDENTIFIER | "(" expression ")"\n               | "super" "." IDENTIFIER ;\n
\n

A1 . 1 . 4Utility rules

\n

In order to keep the above rules a little cleaner, some of the grammar is\nsplit out into a few reused helper rules.

\n
function       → IDENTIFIER "(" parameters? ")" block ;\nparameters     → IDENTIFIER ( "," IDENTIFIER )* ;\narguments      → expression ( "," expression )* ;\n
\n

A1 . 2Lexical Grammar

\n

The lexical grammar is used by the scanner to group characters into tokens.\nWhere the syntax is context free, the lexical grammar is regular—note\nthat there are no recursive rules.

\n
NUMBER         → DIGIT+ ( "." DIGIT+ )? ;\nSTRING         → "\\"" <any char except "\\"">* "\\"" ;\nIDENTIFIER     → ALPHA ( ALPHA | DIGIT )* ;\nALPHA          → "a" ... "z" | "A" ... "Z" | "_" ;\nDIGIT          → "0" ... "9" ;\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/appendix-ii.html", "content": "\n\n\n\nAppendix II · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
A2
\n

Appendix II

\n\n

For your edification, here is the code produced by the little script\nwe built to automate generating the syntax tree classes for jlox.

\n

A2 . 1Expressions

\n

Expressions are the first syntax tree nodes we see, introduced in “Representing\nCode”. The main Expr class defines the visitor\ninterface used to dispatch against the specific expression types, and contains\nthe other expression subclasses as nested classes.

\n
lox/Expr.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.util.List;\n\nabstract class Expr {\n  interface Visitor<R> {\n    R visitAssignExpr(Assign expr);\n    R visitBinaryExpr(Binary expr);\n    R visitCallExpr(Call expr);\n    R visitGetExpr(Get expr);\n    R visitGroupingExpr(Grouping expr);\n    R visitLiteralExpr(Literal expr);\n    R visitLogicalExpr(Logical expr);\n    R visitSetExpr(Set expr);\n    R visitSuperExpr(Super expr);\n    R visitThisExpr(This expr);\n    R visitUnaryExpr(Unary expr);\n    R visitVariableExpr(Variable expr);\n  }\n\n  // Nested Expr classes here...\n\n  abstract <R> R accept(Visitor<R> visitor);\n}\n
\n
lox/Expr.java, create new file
\n\n

A2 . 1 . 1Assign expression

\n

Variable assignment is introduced in “Statements and\nState”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Assign extends Expr {\n    Assign(Token name, Expr value) {\n      this.name = name;\n      this.value = value;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitAssignExpr(this);\n    }\n\n    final Token name;\n    final Expr value;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 2Binary expression

\n

Binary operators are introduced in “Representing\nCode”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Binary extends Expr {\n    Binary(Expr left, Token operator, Expr right) {\n      this.left = left;\n      this.operator = operator;\n      this.right = right;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitBinaryExpr(this);\n    }\n\n    final Expr left;\n    final Token operator;\n    final Expr right;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 3Call expression

\n

Function call expressions are introduced in\n“Functions”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Call extends Expr {\n    Call(Expr callee, Token paren, List<Expr> arguments) {\n      this.callee = callee;\n      this.paren = paren;\n      this.arguments = arguments;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitCallExpr(this);\n    }\n\n    final Expr callee;\n    final Token paren;\n    final List<Expr> arguments;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 4Get expression

\n

Property access, or “get” expressions are introduced in\n“Classes”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Get extends Expr {\n    Get(Expr object, Token name) {\n      this.object = object;\n      this.name = name;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitGetExpr(this);\n    }\n\n    final Expr object;\n    final Token name;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 5Grouping expression

\n

Using parentheses to group expressions is introduced in “Representing\nCode”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Grouping extends Expr {\n    Grouping(Expr expression) {\n      this.expression = expression;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitGroupingExpr(this);\n    }\n\n    final Expr expression;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 6Literal expression

\n

Literal value expressions are introduced in “Representing\nCode”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Literal extends Expr {\n    Literal(Object value) {\n      this.value = value;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitLiteralExpr(this);\n    }\n\n    final Object value;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 7Logical expression

\n

The logical and and or operators are introduced in “Control\nFlow”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Logical extends Expr {\n    Logical(Expr left, Token operator, Expr right) {\n      this.left = left;\n      this.operator = operator;\n      this.right = right;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitLogicalExpr(this);\n    }\n\n    final Expr left;\n    final Token operator;\n    final Expr right;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 8Set expression

\n

Property assignment, or “set” expressions are introduced in\n“Classes”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Set extends Expr {\n    Set(Expr object, Token name, Expr value) {\n      this.object = object;\n      this.name = name;\n      this.value = value;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitSetExpr(this);\n    }\n\n    final Expr object;\n    final Token name;\n    final Expr value;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 9Super expression

\n

The super expression is introduced in\n“Inheritance”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Super extends Expr {\n    Super(Token keyword, Token method) {\n      this.keyword = keyword;\n      this.method = method;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitSuperExpr(this);\n    }\n\n    final Token keyword;\n    final Token method;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 10This expression

\n

The this expression is introduced in “Classes”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class This extends Expr {\n    This(Token keyword) {\n      this.keyword = keyword;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitThisExpr(this);\n    }\n\n    final Token keyword;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 11Unary expression

\n

Unary operators are introduced in “Representing Code”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Unary extends Expr {\n    Unary(Token operator, Expr right) {\n      this.operator = operator;\n      this.right = right;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitUnaryExpr(this);\n    }\n\n    final Token operator;\n    final Expr right;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 1 . 12Variable expression

\n

Variable access expressions are introduced in “Statements and\nState”.

\n
lox/Expr.java
\nnest inside class Expr
\n
  static class Variable extends Expr {\n    Variable(Token name) {\n      this.name = name;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitVariableExpr(this);\n    }\n\n    final Token name;\n  }\n
\n
lox/Expr.java, nest inside class Expr
\n\n

A2 . 2Statements

\n

Statements form a second hierarchy of syntax tree nodes independent of\nexpressions. We add the first couple of them in “Statements and\nState”.

\n
lox/Stmt.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.util.List;\n\nabstract class Stmt {\n  interface Visitor<R> {\n    R visitBlockStmt(Block stmt);\n    R visitClassStmt(Class stmt);\n    R visitExpressionStmt(Expression stmt);\n    R visitFunctionStmt(Function stmt);\n    R visitIfStmt(If stmt);\n    R visitPrintStmt(Print stmt);\n    R visitReturnStmt(Return stmt);\n    R visitVarStmt(Var stmt);\n    R visitWhileStmt(While stmt);\n  }\n\n  // Nested Stmt classes here...\n\n  abstract <R> R accept(Visitor<R> visitor);\n}\n
\n
lox/Stmt.java, create new file
\n\n

A2 . 2 . 1Block statement

\n

The curly-braced block statement that defines a local scope is introduced in\n“Statements and State”.

\n
lox/Stmt.java
\nnest inside class Stmt
\n
  static class Block extends Stmt {\n    Block(List<Stmt> statements) {\n      this.statements = statements;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitBlockStmt(this);\n    }\n\n    final List<Stmt> statements;\n  }\n
\n
lox/Stmt.java, nest inside class Stmt
\n\n

A2 . 2 . 2Class statement

\n

Class declarations are introduced in, unsurprisingly,\n“Classes”.

\n
lox/Stmt.java
\nnest inside class Stmt
\n
  static class Class extends Stmt {\n    Class(Token name,\n          Expr.Variable superclass,\n          List<Stmt.Function> methods) {\n      this.name = name;\n      this.superclass = superclass;\n      this.methods = methods;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitClassStmt(this);\n    }\n\n    final Token name;\n    final Expr.Variable superclass;\n    final List<Stmt.Function> methods;\n  }\n
\n
lox/Stmt.java, nest inside class Stmt
\n\n

A2 . 2 . 3Expression statement

\n

The expression statement is introduced in “Statements and\nState”.

\n
lox/Stmt.java
\nnest inside class Stmt
\n
  static class Expression extends Stmt {\n    Expression(Expr expression) {\n      this.expression = expression;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitExpressionStmt(this);\n    }\n\n    final Expr expression;\n  }\n
\n
lox/Stmt.java, nest inside class Stmt
\n\n

A2 . 2 . 4Function statement

\n

Function declarations are introduced in, you guessed it,\n“Functions”.

\n
lox/Stmt.java
\nnest inside class Stmt
\n
  static class Function extends Stmt {\n    Function(Token name, List<Token> params, List<Stmt> body) {\n      this.name = name;\n      this.params = params;\n      this.body = body;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitFunctionStmt(this);\n    }\n\n    final Token name;\n    final List<Token> params;\n    final List<Stmt> body;\n  }\n
\n
lox/Stmt.java, nest inside class Stmt
\n\n

A2 . 2 . 5If statement

\n

The if statement is introduced in “Control\nFlow”.

\n
lox/Stmt.java
\nnest inside class Stmt
\n
  static class If extends Stmt {\n    If(Expr condition, Stmt thenBranch, Stmt elseBranch) {\n      this.condition = condition;\n      this.thenBranch = thenBranch;\n      this.elseBranch = elseBranch;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitIfStmt(this);\n    }\n\n    final Expr condition;\n    final Stmt thenBranch;\n    final Stmt elseBranch;\n  }\n
\n
lox/Stmt.java, nest inside class Stmt
\n\n

A2 . 2 . 6Print statement

\n

The print statement is introduced in “Statements and\nState”.

\n
lox/Stmt.java
\nnest inside class Stmt
\n
  static class Print extends Stmt {\n    Print(Expr expression) {\n      this.expression = expression;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitPrintStmt(this);\n    }\n\n    final Expr expression;\n  }\n
\n
lox/Stmt.java, nest inside class Stmt
\n\n

A2 . 2 . 7Return statement

\n

You need a function to return from, so return statements are introduced in\n“Functions”.

\n
lox/Stmt.java
\nnest inside class Stmt
\n
  static class Return extends Stmt {\n    Return(Token keyword, Expr value) {\n      this.keyword = keyword;\n      this.value = value;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitReturnStmt(this);\n    }\n\n    final Token keyword;\n    final Expr value;\n  }\n
\n
lox/Stmt.java, nest inside class Stmt
\n\n

A2 . 2 . 8Variable statement

\n

Variable declarations are introduced in “Statements and\nState”.

\n
lox/Stmt.java
\nnest inside class Stmt
\n
  static class Var extends Stmt {\n    Var(Token name, Expr initializer) {\n      this.name = name;\n      this.initializer = initializer;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitVarStmt(this);\n    }\n\n    final Token name;\n    final Expr initializer;\n  }\n
\n
lox/Stmt.java, nest inside class Stmt
\n\n

A2 . 2 . 9While statement

\n

The while statement is introduced in “Control\nFlow”.

\n
lox/Stmt.java
\nnest inside class Stmt
\n
  static class While extends Stmt {\n    While(Expr condition, Stmt body) {\n      this.condition = condition;\n      this.body = body;\n    }\n\n    @Override\n    <R> R accept(Visitor<R> visitor) {\n      return visitor.visitWhileStmt(this);\n    }\n\n    final Expr condition;\n    final Stmt body;\n  }\n
\n
lox/Stmt.java, nest inside class Stmt
\n\n\n\n
\n\n
\n\n\n" }, { "path": "site/backmatter.html", "content": "\n\n\n\nBackmatter · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n\n\n
\n\n\n" }, { "path": "site/calls-and-functions.html", "content": "\n\n\n\nCalls and Functions · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
24
\n

Calls and Functions

\n\n
\n

Any problem in computer science can be solved with another level of\nindirection. Except for the problem of too many layers of indirection.

\n

David Wheeler

\n
\n

This chapter is a beast. I try to break features into bite-sized pieces, but\nsometimes you gotta swallow the whole meal. Our next\ntask is functions. We could start with only function declarations, but that’s\nnot very useful when you can’t call them. We could do calls, but there’s nothing\nto call. And all of the runtime support needed in the VM to support both of\nthose isn’t very rewarding if it isn’t hooked up to anything you can see. So\nwe’re going to do it all. It’s a lot, but we’ll feel good when we’re done.

\n\n

24 . 1Function Objects

\n

The most interesting structural change in the VM is around the stack. We already\nhave a stack for local variables and temporaries, so we’re partway there. But\nwe have no notion of a call stack. Before we can make much progress, we’ll\nhave to fix that. But first, let’s write some code. I always feel better once I\nstart moving. We can’t do much without having some kind of representation for\nfunctions, so we’ll start there. From the VM’s perspective, what is a function?

\n

A function has a body that can be executed, so that means some bytecode. We\ncould compile the entire program and all of its function declarations into one\nbig monolithic Chunk. Each function would have a pointer to the first\ninstruction of its code inside the Chunk.

\n

This is roughly how compilation to native code works where you end up with one\nsolid blob of machine code. But for our bytecode VM, we can do something a\nlittle higher level. I think a cleaner model is to give each function its own\nChunk. We’ll want some other metadata too, so let’s go ahead and stuff it all in\na struct now.

\n
  struct Obj* next;\n};\n
object.h
\nadd after struct Obj
\n
\n\ntypedef struct {\n  Obj obj;\n  int arity;\n  Chunk chunk;\n  ObjString* name;\n} ObjFunction;\n
\n\nstruct ObjString {\n
\n
object.h, add after struct Obj
\n\n

Functions are first class in Lox, so they need to be actual Lox objects. Thus\nObjFunction has the same Obj header that all object types share. The arity\nfield stores the number of parameters the function expects. Then, in addition to\nthe chunk, we store the function’s name. That will be\nhandy for reporting readable runtime errors.

\n\n

This is the first time the “object” module has needed to reference Chunk, so we\nget an include.

\n
#include "common.h"\n
object.h
\n
#include "chunk.h"\n
#include "value.h"\n
\n
object.h
\n\n

Like we did with strings, we define some accessories to make Lox functions\neasier to work with in C. Sort of a poor man’s object orientation. First, we’ll\ndeclare a C function to create a new Lox function.

\n
  uint32_t hash;\n};\n\n
object.h
\nadd after struct ObjString
\n
ObjFunction* newFunction();\n
ObjString* takeString(char* chars, int length);\n
\n
object.h, add after struct ObjString
\n\n

The implementation is over here:

\n
object.c
\nadd after allocateObject()
\n
ObjFunction* newFunction() {\n  ObjFunction* function = ALLOCATE_OBJ(ObjFunction, OBJ_FUNCTION);\n  function->arity = 0;\n  function->name = NULL;\n  initChunk(&function->chunk);\n  return function;\n}\n
\n
object.c, add after allocateObject()
\n\n

We use our friend ALLOCATE_OBJ() to allocate memory and initialize the\nobject’s header so that the VM knows what type of object it is. Instead of\npassing in arguments to initialize the function like we did with ObjString, we\nset the function up in a sort of blank state—zero arity, no name, and no\ncode. That will get filled in later after the function is created.

\n

Since we have a new kind of object, we need a new object type in the enum.

\n
typedef enum {\n
object.h
\nin enum ObjType
\n
  OBJ_FUNCTION,\n
  OBJ_STRING,\n} ObjType;\n
\n
object.h, in enum ObjType
\n\n

When we’re done with a function object, we must return the bits it borrowed back\nto the operating system.

\n
  switch (object->type) {\n
memory.c
\nin freeObject()
\n
    case OBJ_FUNCTION: {\n      ObjFunction* function = (ObjFunction*)object;\n      freeChunk(&function->chunk);\n      FREE(ObjFunction, object);\n      break;\n    }\n
    case OBJ_STRING: {\n
\n
memory.c, in freeObject()
\n\n

This switch case is responsible for freeing the\nObjFunction itself as well as any other memory it owns. Functions own their\nchunk, so we call Chunk’s destructor-like function.

\n\n

Lox lets you print any object, and functions are first-class objects, so we\nneed to handle them too.

\n
  switch (OBJ_TYPE(value)) {\n
object.c
\nin printObject()
\n
    case OBJ_FUNCTION:\n      printFunction(AS_FUNCTION(value));\n      break;\n
    case OBJ_STRING:\n
\n
object.c, in printObject()
\n\n

This calls out to:

\n
object.c
\nadd after copyString()
\n
static void printFunction(ObjFunction* function) {\n  printf("<fn %s>", function->name->chars);\n}\n
\n
object.c, add after copyString()
\n\n

Since a function knows its name, it may as well say it.

\n

Finally, we have a couple of macros for converting values to functions. First,\nmake sure your value actually is a function.

\n
#define OBJ_TYPE(value)        (AS_OBJ(value)->type)\n\n
object.h
\n
#define IS_FUNCTION(value)     isObjType(value, OBJ_FUNCTION)\n
#define IS_STRING(value)       isObjType(value, OBJ_STRING)\n
\n
object.h
\n\n

Assuming that evaluates to true, you can then safely cast the Value to an\nObjFunction pointer using this:

\n
#define IS_STRING(value)       isObjType(value, OBJ_STRING)\n\n
object.h
\n
#define AS_FUNCTION(value)     ((ObjFunction*)AS_OBJ(value))\n
#define AS_STRING(value)       ((ObjString*)AS_OBJ(value))\n
\n
object.h
\n\n

With that, our object model knows how to represent functions. I’m feeling warmed\nup now. You ready for something a little harder?

\n

24 . 2Compiling to Function Objects

\n

Right now, our compiler assumes it is always compiling to one single chunk. With\neach function’s code living in separate chunks, that gets more complex. When the\ncompiler reaches a function declaration, it needs to emit code into the\nfunction’s chunk when compiling its body. At the end of the function body, the\ncompiler needs to return to the previous chunk it was working with.

\n

That’s fine for code inside function bodies, but what about code that isn’t? The\n“top level” of a Lox program is also imperative code and we need a chunk to\ncompile that into. We can simplify the compiler and VM by placing that top-level\ncode inside an automatically defined function too. That way, the compiler is\nalways within some kind of function body, and the VM always runs code by\ninvoking a function. It’s as if the entire program is wrapped inside an implicit main() function.

\n\n

Before we get to user-defined functions, then, let’s do the reorganization to\nsupport that implicit top-level function. It starts with the Compiler struct.\nInstead of pointing directly to a Chunk that the compiler writes to, it instead\nhas a reference to the function object being built.

\n
typedef struct {\n
compiler.c
\nin struct Compiler
\n
  ObjFunction* function;\n  FunctionType type;\n\n
  Local locals[UINT8_COUNT];\n
\n
compiler.c, in struct Compiler
\n\n

We also have a little FunctionType enum. This lets the compiler tell when it’s\ncompiling top-level code versus the body of a function. Most of the compiler\ndoesn’t care about this—that’s why it’s a useful abstraction—but in one or\ntwo places the distinction is meaningful. We’ll get to one later.

\n
compiler.c
\nadd after struct Local
\n
typedef enum {\n  TYPE_FUNCTION,\n  TYPE_SCRIPT\n} FunctionType;\n
\n
compiler.c, add after struct Local
\n\n

Every place in the compiler that was writing to the Chunk now needs to go\nthrough that function pointer. Fortunately, many chapters ago, we encapsulated access to the chunk in the\ncurrentChunk() function. We only need to fix that and the rest of the compiler\nis happy.

\n\n
Compiler* current = NULL;\n
compiler.c
\nadd after variable current
\nreplace 5 lines
\n
\n\nstatic Chunk* currentChunk() {\n  return &current->function->chunk;\n}\n
\n\nstatic void errorAt(Token* token, const char* message) {\n
\n
compiler.c, add after variable current, replace 5 lines
\n\n

The current chunk is always the chunk owned by the function we’re in the middle\nof compiling. Next, we need to actually create that function. Previously, the VM\npassed a Chunk to the compiler which filled it with code. Instead, the compiler\nwill create and return a function that contains the compiled top-level code—which is all we support right now—of the user’s program.

\n

24 . 2 . 1Creating functions at compile time

\n

We start threading this through in compile(), which is the main entry point\ninto the compiler.

\n
  Compiler compiler;\n
compiler.c
\nin compile()
\nreplace 2 lines
\n
  initCompiler(&compiler, TYPE_SCRIPT);\n
\n\n  parser.hadError = false;\n
\n
compiler.c, in compile(), replace 2 lines
\n\n

There are a bunch of changes in how the compiler is initialized. First, we\ninitialize the new Compiler fields.

\n
compiler.c
\nfunction initCompiler()
\nreplace 1 line
\n
static void initCompiler(Compiler* compiler, FunctionType type) {\n  compiler->function = NULL;\n  compiler->type = type;\n
  compiler->localCount = 0;\n
\n
compiler.c, function initCompiler(), replace 1 line
\n\n

Then we allocate a new function object to compile into.

\n
  compiler->scopeDepth = 0;\n
compiler.c
\nin initCompiler()
\n
  compiler->function = newFunction();\n
  current = compiler;\n
\n
compiler.c, in initCompiler()
\n\n

\n\n

Creating an ObjFunction in the compiler might seem a little strange. A function\nobject is the runtime representation of a function, but here we are creating\nit at compile time. The way to think of it is that a function is similar to a\nstring or number literal. It forms a bridge between the compile time and runtime\nworlds. When we get to function declarations, those really are literals—they are a notation that produces values of a built-in type. So the compiler creates function objects during compilation.\nThen, at runtime, they are simply invoked.

\n\n

Here is another strange piece of code:

\n
  current = compiler;\n
compiler.c
\nin initCompiler()
\n
\n\n  Local* local = &current->locals[current->localCount++];\n  local->depth = 0;\n  local->name.start = "";\n  local->name.length = 0;\n
}\n
\n
compiler.c, in initCompiler()
\n\n

Remember that the compiler’s locals array keeps track of which stack slots are\nassociated with which local variables or temporaries. From now on, the compiler\nimplicitly claims stack slot zero for the VM’s own internal use. We give it an\nempty name so that the user can’t write an identifier that refers to it. I’ll\nexplain what this is about when it becomes useful.

\n

That’s the initialization side. We also need a couple of changes on the other\nend when we finish compiling some code.

\n
compiler.c
\nfunction endCompiler()
\nreplace 1 line
\n
static ObjFunction* endCompiler() {\n
  emitReturn();\n
\n
compiler.c, function endCompiler(), replace 1 line
\n\n

Previously, when interpret() called into the compiler, it passed in a Chunk to\nbe written to. Now that the compiler creates the function object itself, we\nreturn that function. We grab it from the current compiler here:

\n
  emitReturn();\n
compiler.c
\nin endCompiler()
\n
  ObjFunction* function = current->function;\n\n
#ifdef DEBUG_PRINT_CODE\n
\n
compiler.c, in endCompiler()
\n\n

And then return it to compile() like so:

\n
#endif\n
compiler.c
\nin endCompiler()
\n
\n\n  return function;\n
}\n
\n
compiler.c, in endCompiler()
\n\n

Now is a good time to make another tweak in this function. Earlier, we added\nsome diagnostic code to have the VM dump the disassembled bytecode so we could\ndebug the compiler. We should fix that to keep working now that the generated\nchunk is wrapped in a function.

\n
#ifdef DEBUG_PRINT_CODE\n  if (!parser.hadError) {\n
compiler.c
\nin endCompiler()
\nreplace 1 line
\n
    disassembleChunk(currentChunk(), function->name != NULL\n        ? function->name->chars : "<script>");\n
  }\n#endif\n
\n
compiler.c, in endCompiler(), replace 1 line
\n\n

Notice the check in here to see if the function’s name is NULL? User-defined\nfunctions have names, but the implicit function we create for the top-level code\ndoes not, and we need to handle that gracefully even in our own diagnostic code.\nSpeaking of which:

\n
static void printFunction(ObjFunction* function) {\n
object.c
\nin printFunction()
\n
  if (function->name == NULL) {\n    printf("<script>");\n    return;\n  }\n
  printf("<fn %s>", function->name->chars);\n
\n
object.c, in printFunction()
\n\n

There’s no way for a user to get a reference to the top-level function and try\nto print it, but our DEBUG_TRACE_EXECUTION diagnostic code that prints the entire stack can and does.

\n\n

Bumping up a level to compile(), we adjust its signature.

\n
#include "vm.h"\n\n
compiler.h
\nfunction compile()
\nreplace 1 line
\n
ObjFunction* compile(const char* source);\n
\n\n#endif\n
\n
compiler.h, function compile(), replace 1 line
\n\n

Instead of taking a chunk, now it returns a function. Over in the\nimplementation:

\n
compiler.c
\nfunction compile()
\nreplace 1 line
\n
ObjFunction* compile(const char* source) {\n
  initScanner(source);\n
\n
compiler.c, function compile(), replace 1 line
\n\n

Finally we get to some actual code. We change the very end of the function to\nthis:

\n
  while (!match(TOKEN_EOF)) {\n    declaration();\n  }\n\n
compiler.c
\nin compile()
\nreplace 2 lines
\n
  ObjFunction* function = endCompiler();\n  return parser.hadError ? NULL : function;\n
}\n
\n
compiler.c, in compile(), replace 2 lines
\n\n

We get the function object from the compiler. If there were no compile errors,\nwe return it. Otherwise, we signal an error by returning NULL. This way, the\nVM doesn’t try to execute a function that may contain invalid bytecode.

\n

Eventually, we will update interpret() to handle the new declaration of\ncompile(), but first we have some other changes to make.

\n

24 . 3Call Frames

\n

It’s time for a big conceptual leap. Before we can implement function\ndeclarations and calls, we need to get the VM ready to handle them. There are\ntwo main problems we need to worry about:

\n

24 . 3 . 1Allocating local variables

\n

The compiler allocates stack slots for local variables. How should that work\nwhen the set of local variables in a program is distributed across multiple\nfunctions?

\n

One option would be to keep them totally separate. Each function would get its\nown dedicated set of slots in the VM stack that it would own forever, even when the function isn’t being called. Each\nlocal variable in the entire program would have a bit of memory in the VM that\nit keeps to itself.

\n\n

Believe it or not, early programming language implementations worked this way.\nThe first Fortran compilers statically allocated memory for each variable. The\nobvious problem is that it’s really inefficient. Most functions are not in the\nmiddle of being called at any point in time, so sitting on unused memory for\nthem is wasteful.

\n

The more fundamental problem, though, is recursion. With recursion, you can be\n“in” multiple calls to the same function at the same time. Each needs its own memory for its local variables. In jlox, we solved\nthis by dynamically allocating memory for an environment each time a function\nwas called or a block entered. In clox, we don’t want that kind of performance\ncost on every function call.

\n\n

Instead, our solution lies somewhere between Fortran’s static allocation and\njlox’s dynamic approach. The value stack in the VM works on the observation that\nlocal variables and temporaries behave in a last-in first-out fashion.\nFortunately for us, that’s still true even when you add function calls into the\nmix. Here’s an example:

\n
fun first() {\n  var a = 1;\n  second();\n  var b = 2;\n}\n\nfun second() {\n  var c = 3;\n  var d = 4;\n}\n\nfirst();\n
\n

Step through the program and look at which variables are in memory at each point\nin time:

\"Tracing\n

As execution flows through the two calls, every local variable obeys the\nprinciple that any variable declared after it will be discarded before the first\nvariable needs to be. This is true even across calls. We know we’ll be done with\nc and d before we are done with a. It seems we should be able to allocate\nlocal variables on the VM’s value stack.

\n

Ideally, we still determine where on the stack each variable will go at\ncompile time. That keeps the bytecode instructions for working with variables\nsimple and fast. In the above example, we could imagine doing so in a straightforward way, but that\ndoesn’t always work out. Consider:

\n\n
fun first() {\n  var a = 1;\n  second();\n  var b = 2;\n  second();\n}\n\nfun second() {\n  var c = 3;\n  var d = 4;\n}\n\nfirst();\n
\n

In the first call to second(), c and d would go into slots 1 and 2. But in\nthe second call, we need to have made room for b, so c and d need to be in\nslots 2 and 3. Thus the compiler can’t pin down an exact slot for each local\nvariable across function calls. But within a given function, the relative\nlocations of each local variable are fixed. Variable d is always in the slot\nright after c. This is the key insight.

\n

When a function is called, we don’t know where the top of the stack will be\nbecause it can be called from different contexts. But, wherever that top happens\nto be, we do know where all of the function’s local variables will be relative\nto that starting point. So, like many problems, we solve our allocation problem\nwith a level of indirection.

\n

At the beginning of each function call, the VM records the location of the first\nslot where that function’s own locals begin. The instructions for working with\nlocal variables access them by a slot index relative to that, instead of\nrelative to the bottom of the stack like they do today. At compile time, we\ncalculate those relative slots. At runtime, we convert that relative slot to an\nabsolute stack index by adding the function call’s starting slot.

\n

It’s as if the function gets a “window” or “frame” within the larger stack where\nit can store its locals. The position of the call frame is determined at\nruntime, but within and relative to that region, we know where to find things.

\"The\n

The historical name for this recorded location where the function’s locals start\nis a frame pointer because it points to the beginning of the function’s call\nframe. Sometimes you hear base pointer, because it points to the base stack\nslot on top of which all of the function’s variables live.

\n

That’s the first piece of data we need to track. Every time we call a function,\nthe VM determines the first stack slot where that function’s variables begin.

\n

24 . 3 . 2Return addresses

\n

Right now, the VM works its way through the instruction stream by incrementing\nthe ip field. The only interesting behavior is around control flow\ninstructions which offset the ip by larger amounts. Calling a function is\npretty straightforward—simply set ip to point to the first instruction in\nthat function’s chunk. But what about when the function is done?

\n

The VM needs to return back to the chunk where the\nfunction was called from and resume execution at the instruction immediately\nafter the call. Thus, for each function call, we need to track where we jump\nback to when the call completes. This is called a return address because\nit’s the address of the instruction that the VM returns to after the call.

\n

Again, thanks to recursion, there may be multiple return addresses for a single\nfunction, so this is a property of each invocation and not the function\nitself.

\n\n

24 . 3 . 3The call stack

\n

So for each live function invocation—each call that hasn’t returned yet—we\nneed to track where on the stack that function’s locals begin, and where the\ncaller should resume. We’ll put this, along with some other stuff, in a new\nstruct.

\n
#define STACK_MAX 256\n
vm.h
\n
\n\ntypedef struct {\n  ObjFunction* function;\n  uint8_t* ip;\n  Value* slots;\n} CallFrame;\n
\n\ntypedef struct {\n
\n
vm.h
\n\n

A CallFrame represents a single ongoing function call. The slots field points\ninto the VM’s value stack at the first slot that this function can use. I gave\nit a plural name because—thanks to C’s weird “pointers are sort of arrays”\nthing—we’ll treat it like an array.

\n

The implementation of return addresses is a little different from what I\ndescribed above. Instead of storing the return address in the callee’s frame,\nthe caller stores its own ip. When we return from a function, the VM will jump\nto the ip of the caller’s CallFrame and resume from there.

\n

I also stuffed a pointer to the function being called in here. We’ll use that to\nlook up constants and for a few other things.

\n

Each time a function is called, we create one of these structs. We could dynamically allocate them on the heap, but that’s slow.\nFunction calls are a core operation, so they need to be as fast as possible.\nFortunately, we can make the same observation we made for variables: function\ncalls have stack semantics. If first() calls second(), the call to\nsecond() will complete before first() does.

\n\n

So over in the VM, we create an array of these CallFrame structs up front and\ntreat it as a stack, like we do with the value array.

\n
typedef struct {\n
vm.h
\nin struct VM
\nreplace 2 lines
\n
  CallFrame frames[FRAMES_MAX];\n  int frameCount;\n\n
  Value stack[STACK_MAX];\n
\n
vm.h, in struct VM, replace 2 lines
\n\n

This array replaces the chunk and ip fields we used to have directly in the\nVM. Now each CallFrame has its own ip and its own pointer to the ObjFunction\nthat it’s executing. From there, we can get to the function’s chunk.

\n

The new frameCount field in the VM stores the current height of the CallFrame\nstack—the number of ongoing function calls. To keep clox simple, the array’s\ncapacity is fixed. This means, as in many language implementations, there is a\nmaximum call depth we can handle. For clox, it’s defined here:

\n
#include "value.h"\n\n
vm.h
\nreplace 1 line
\n
#define FRAMES_MAX 64\n#define STACK_MAX (FRAMES_MAX * UINT8_COUNT)\n
\n\ntypedef struct {\n
\n
vm.h, replace 1 line
\n\n

We also redefine the value stack’s size in terms of\nthat to make sure we have plenty of stack slots even in very deep call trees.\nWhen the VM starts up, the CallFrame stack is empty.

\n\n
  vm.stackTop = vm.stack;\n
vm.c
\nin resetStack()
\n
  vm.frameCount = 0;\n
}\n
\n
vm.c, in resetStack()
\n\n

The “vm.h” header needs access to ObjFunction, so we add an include.

\n
#define clox_vm_h\n\n
vm.h
\nreplace 1 line
\n
#include "object.h"\n
#include "table.h"\n
\n
vm.h, replace 1 line
\n\n

Now we’re ready to move over to the VM’s implementation file. We’ve got some\ngrunt work ahead of us. We’ve moved ip out of the VM struct and into\nCallFrame. We need to fix every line of code in the VM that touches ip to\nhandle that. Also, the instructions that access local variables by stack slot\nneed to be updated to do so relative to the current CallFrame’s slots field.

\n

We’ll start at the top and plow through it.

\n
static InterpretResult run() {\n
vm.c
\nin run()
\nreplace 4 lines
\n
  CallFrame* frame = &vm.frames[vm.frameCount - 1];\n\n#define READ_BYTE() (*frame->ip++)\n\n#define READ_SHORT() \\\n    (frame->ip += 2, \\\n    (uint16_t)((frame->ip[-2] << 8) | frame->ip[-1]))\n\n#define READ_CONSTANT() \\\n    (frame->function->chunk.constants.values[READ_BYTE()])\n\n
#define READ_STRING() AS_STRING(READ_CONSTANT())\n
\n
vm.c, in run(), replace 4 lines
\n\n

First, we store the current topmost CallFrame in a local variable inside the main bytecode execution function.\nThen we replace the bytecode access macros with versions that access ip\nthrough that variable.

\n\n

Now onto each instruction that needs a little tender loving care.

\n
      case OP_GET_LOCAL: {\n        uint8_t slot = READ_BYTE();\n
vm.c
\nin run()
\nreplace 1 line
\n
        push(frame->slots[slot]);\n
        break;\n
\n
vm.c, in run(), replace 1 line
\n\n

Previously, OP_GET_LOCAL read the given local slot directly from the VM’s\nstack array, which meant it indexed the slot starting from the bottom of the\nstack. Now, it accesses the current frame’s slots array, which means it\naccesses the given numbered slot relative to the beginning of that frame.

\n

Setting a local variable works the same way.

\n
      case OP_SET_LOCAL: {\n        uint8_t slot = READ_BYTE();\n
vm.c
\nin run()
\nreplace 1 line
\n
        frame->slots[slot] = peek(0);\n
        break;\n
\n
vm.c, in run(), replace 1 line
\n\n

The jump instructions used to modify the VM’s ip field. Now, they do the same\nfor the current frame’s ip.

\n
      case OP_JUMP: {\n        uint16_t offset = READ_SHORT();\n
vm.c
\nin run()
\nreplace 1 line
\n
        frame->ip += offset;\n
        break;\n
\n
vm.c, in run(), replace 1 line
\n\n

Same with the conditional jump:

\n
      case OP_JUMP_IF_FALSE: {\n        uint16_t offset = READ_SHORT();\n
vm.c
\nin run()
\nreplace 1 line
\n
        if (isFalsey(peek(0))) frame->ip += offset;\n
        break;\n
\n
vm.c, in run(), replace 1 line
\n\n

And our backward-jumping loop instruction:

\n
      case OP_LOOP: {\n        uint16_t offset = READ_SHORT();\n
vm.c
\nin run()
\nreplace 1 line
\n
        frame->ip -= offset;\n
        break;\n
\n
vm.c, in run(), replace 1 line
\n\n

We have some diagnostic code that prints each instruction as it executes to help\nus debug our VM. That needs to work with the new structure too.

\n
    printf("\\n");\n
vm.c
\nin run()
\nreplace 2 lines
\n
    disassembleInstruction(&frame->function->chunk,\n        (int)(frame->ip - frame->function->chunk.code));\n
#endif\n
\n
vm.c, in run(), replace 2 lines
\n\n

Instead of passing in the VM’s chunk and ip fields, now we read from the\ncurrent CallFrame.

\n

You know, that wasn’t too bad, actually. Most instructions just use the macros\nso didn’t need to be touched. Next, we jump up a level to the code that calls\nrun().

\n
InterpretResult interpret(const char* source) {\n
vm.c
\nin interpret()
\nreplace 10 lines
\n
  ObjFunction* function = compile(source);\n  if (function == NULL) return INTERPRET_COMPILE_ERROR;\n\n  push(OBJ_VAL(function));\n  CallFrame* frame = &vm.frames[vm.frameCount++];\n  frame->function = function;\n  frame->ip = function->chunk.code;\n  frame->slots = vm.stack;\n
\n\n  InterpretResult result = run();\n
\n
vm.c, in interpret(), replace 10 lines
\n\n

We finally get to wire up our earlier compiler changes to the back-end changes\nwe just made. First, we pass the source code to the compiler. It returns us a\nnew ObjFunction containing the compiled top-level code. If we get NULL back,\nit means there was some compile-time error which the compiler has already\nreported. In that case, we bail out since we can’t run anything.

\n

Otherwise, we store the function on the stack and prepare an initial CallFrame\nto execute its code. Now you can see why the compiler sets aside stack slot zero—that stores the function being called. In the new CallFrame, we point to the\nfunction, initialize its ip to point to the beginning of the function’s\nbytecode, and set up its stack window to start at the very bottom of the VM’s\nvalue stack.

\n

This gets the interpreter ready to start executing code. After finishing, the VM\nused to free the hardcoded chunk. Now that the ObjFunction owns that code, we\ndon’t need to do that anymore, so the end of interpret() is simply this:

\n
  frame->slots = vm.stack;\n\n
vm.c
\nin interpret()
\nreplace 4 lines
\n
  return run();\n
}\n
\n
vm.c, in interpret(), replace 4 lines
\n\n

The last piece of code referring to the old VM fields is runtimeError(). We’ll\nrevisit that later in the chapter, but for now let’s change it to this:

\n
  fputs("\\n", stderr);\n\n
vm.c
\nin runtimeError()
\nreplace 2 lines
\n
  CallFrame* frame = &vm.frames[vm.frameCount - 1];\n  size_t instruction = frame->ip - frame->function->chunk.code - 1;\n  int line = frame->function->chunk.lines[instruction];\n
  fprintf(stderr, "[line %d] in script\\n", line);\n
\n
vm.c, in runtimeError(), replace 2 lines
\n\n

Instead of reading the chunk and ip directly from the VM, it pulls those from\nthe topmost CallFrame on the stack. That should get the function working again\nand behaving as it did before.

\n

Assuming we did all of that correctly, we got clox back to a runnable\nstate. Fire it up and it does . . . exactly what it did before. We haven’t added\nany new features yet, so this is kind of a let down. But all of the\ninfrastructure is there and ready for us now. Let’s take advantage of it.

\n

24 . 4Function Declarations

\n

Before we can do call expressions, we need something to call, so we’ll do\nfunction declarations first. The fun starts with a\nkeyword.

\n\n
static void declaration() {\n
compiler.c
\nin declaration()
\nreplace 1 line
\n
  if (match(TOKEN_FUN)) {\n    funDeclaration();\n  } else if (match(TOKEN_VAR)) {\n
    varDeclaration();\n
\n
compiler.c, in declaration(), replace 1 line
\n\n

That passes control to here:

\n
compiler.c
\nadd after block()
\n
static void funDeclaration() {\n  uint8_t global = parseVariable("Expect function name.");\n  markInitialized();\n  function(TYPE_FUNCTION);\n  defineVariable(global);\n}\n
\n
compiler.c, add after block()
\n\n

Functions are first-class values, and a function declaration simply creates and\nstores one in a newly declared variable. So we parse the name just like any\nother variable declaration. A function declaration at the top level will bind\nthe function to a global variable. Inside a block or other function, a function\ndeclaration creates a local variable.

\n

In an earlier chapter, I explained how variables get defined in two\nstages. This ensures you can’t access a variable’s value inside the\nvariable’s own initializer. That would be bad because the variable doesn’t\nhave a value yet.

\n

Functions don’t suffer from this problem. It’s safe for a function to refer to\nits own name inside its body. You can’t call the function and execute the body\nuntil after it’s fully defined, so you’ll never see the variable in an\nuninitialized state. Practically speaking, it’s useful to allow this in order to\nsupport recursive local functions.

\n

To make that work, we mark the function declaration’s variable “initialized” as\nsoon as we compile the name, before we compile the body. That way the name can\nbe referenced inside the body without generating an error.

\n

We do need one check, though.

\n
static void markInitialized() {\n
compiler.c
\nin markInitialized()
\n
  if (current->scopeDepth == 0) return;\n
  current->locals[current->localCount - 1].depth =\n
\n
compiler.c, in markInitialized()
\n\n

Before, we called markInitialized() only when we already knew we were in a\nlocal scope. Now, a top-level function declaration will also call this function.\nWhen that happens, there is no local variable to mark initialized—the\nfunction is bound to a global variable.

\n

Next, we compile the function itself—its parameter list and block body. For\nthat, we use a separate helper function. That helper generates code that\nleaves the resulting function object on top of the stack. After that, we call\ndefineVariable() to store that function back into the variable we declared for\nit.

\n

I split out the code to compile the parameters and body because we’ll reuse it\nlater for parsing method declarations inside classes. Let’s build it\nincrementally, starting with this:

\n
compiler.c
\nadd after block()
\n
static void function(FunctionType type) {\n  Compiler compiler;\n  initCompiler(&compiler, type);\n  beginScope(); \n\n  consume(TOKEN_LEFT_PAREN, "Expect '(' after function name.");\n  consume(TOKEN_RIGHT_PAREN, "Expect ')' after parameters.");\n  consume(TOKEN_LEFT_BRACE, "Expect '{' before function body.");\n  block();\n\n  ObjFunction* function = endCompiler();\n  emitBytes(OP_CONSTANT, makeConstant(OBJ_VAL(function)));\n}\n
\n
compiler.c, add after block()
\n\n\n

For now, we won’t worry about parameters. We parse an empty pair of parentheses\nfollowed by the body. The body starts with a left curly brace, which we parse\nhere. Then we call our existing block() function, which knows how to compile\nthe rest of a block including the closing brace.

\n

24 . 4 . 1A stack of compilers

\n

The interesting parts are the compiler stuff at the top and bottom. The Compiler\nstruct stores data like which slots are owned by which local variables, how many\nblocks of nesting we’re currently in, etc. All of that is specific to a single\nfunction. But now the front end needs to handle compiling multiple functions\nnested within each other.

\n\n

The trick for managing that is to create a separate Compiler for each function\nbeing compiled. When we start compiling a function declaration, we create a new\nCompiler on the C stack and initialize it. initCompiler() sets that Compiler\nto be the current one. Then, as we compile the body, all of the functions that\nemit bytecode write to the chunk owned by the new Compiler’s function.

\n

After we reach the end of the function’s block body, we call endCompiler().\nThat yields the newly compiled function object, which we store as a constant in\nthe surrounding function’s constant table. But, wait, how do we get back to\nthe surrounding function? We lost it when initCompiler() overwrote the current\ncompiler pointer.

\n

We fix that by treating the series of nested Compiler structs as a stack. Unlike\nthe Value and CallFrame stacks in the VM, we won’t use an array. Instead, we use\na linked list. Each Compiler points back to the Compiler for the function that\nencloses it, all the way back to the root Compiler for the top-level code.

\n
} FunctionType;\n\n
compiler.c
\nadd after enum FunctionType
\nreplace 1 line
\n
typedef struct Compiler {\n  struct Compiler* enclosing;\n
  ObjFunction* function;\n
\n
compiler.c, add after enum FunctionType, replace 1 line
\n\n

Inside the Compiler struct, we can’t reference the Compiler typedef since that\ndeclaration hasn’t finished yet. Instead, we give a name to the struct itself\nand use that for the field’s type. C is weird.

\n

When initializing a new Compiler, we capture the about-to-no-longer-be-current\none in that pointer.

\n
static void initCompiler(Compiler* compiler, FunctionType type) {\n
compiler.c
\nin initCompiler()
\n
  compiler->enclosing = current;\n
  compiler->function = NULL;\n
\n
compiler.c, in initCompiler()
\n\n

Then when a Compiler finishes, it pops itself off the stack by restoring the\nprevious compiler to be the new current one.

\n
#endif\n\n
compiler.c
\nin endCompiler()
\n
  current = current->enclosing;\n
  return function;\n
\n
compiler.c, in endCompiler()
\n\n

Note that we don’t even need to dynamically\nallocate the Compiler structs. Each is stored as a local variable in the C stack—either in compile() or function(). The linked list of Compilers threads\nthrough the C stack. The reason we can get an unbounded number of them is\nbecause our compiler uses recursive descent, so function() ends up calling\nitself recursively when you have nested function declarations.

\n\n

24 . 4 . 2Function parameters

\n

Functions aren’t very useful if you can’t pass arguments to them, so let’s do\nparameters next.

\n
  consume(TOKEN_LEFT_PAREN, "Expect '(' after function name.");\n
compiler.c
\nin function()
\n
  if (!check(TOKEN_RIGHT_PAREN)) {\n    do {\n      current->function->arity++;\n      if (current->function->arity > 255) {\n        errorAtCurrent("Can't have more than 255 parameters.");\n      }\n      uint8_t constant = parseVariable("Expect parameter name.");\n      defineVariable(constant);\n    } while (match(TOKEN_COMMA));\n  }\n
  consume(TOKEN_RIGHT_PAREN, "Expect ')' after parameters.");\n
\n
compiler.c, in function()
\n\n

Semantically, a parameter is simply a local variable declared in the outermost\nlexical scope of the function body. We get to use the existing compiler support\nfor declaring named local variables to parse and compile parameters. Unlike\nlocal variables, which have initializers, there’s no code here to initialize the\nparameter’s value. We’ll see how they are initialized later when we do argument\npassing in function calls.

\n

While we’re at it, we note the function’s arity by counting how many parameters\nwe parse. The other piece of metadata we store with a function is its name. When\ncompiling a function declaration, we call initCompiler() right after we parse\nthe function’s name. That means we can grab the name right then from the\nprevious token.

\n
  current = compiler;\n
compiler.c
\nin initCompiler()
\n
  if (type != TYPE_SCRIPT) {\n    current->function->name = copyString(parser.previous.start,\n                                         parser.previous.length);\n  }\n
\n\n  Local* local = &current->locals[current->localCount++];\n
\n
compiler.c, in initCompiler()
\n\n

Note that we’re careful to create a copy of the name string. Remember, the\nlexeme points directly into the original source code string. That string may get\nfreed once the code is finished compiling. The function object we create in the\ncompiler outlives the compiler and persists until runtime. So it needs its own\nheap-allocated name string that it can keep around.

\n

Rad. Now we can compile function declarations, like this:

\n
fun areWeHavingItYet() {\n  print "Yes we are!";\n}\n\nprint areWeHavingItYet;\n
\n

We just can’t do anything useful with them.

\n\n

24 . 5Function Calls

\n

By the end of this section, we’ll start to see some interesting behavior. The\nnext step is calling functions. We don’t usually think of it this way, but a\nfunction call expression is kind of an infix ( operator. You have a\nhigh-precedence expression on the left for the thing being called—usually\njust a single identifier. Then the ( in the middle, followed by the argument\nexpressions separated by commas, and a final ) to wrap it up at the end.

\n

That odd grammatical perspective explains how to hook the syntax into our\nparsing table.

\n
ParseRule rules[] = {\n
compiler.c
\nadd after unary()
\nreplace 1 line
\n
  [TOKEN_LEFT_PAREN]    = {grouping, call,   PREC_CALL},\n
  [TOKEN_RIGHT_PAREN]   = {NULL,     NULL,   PREC_NONE},\n
\n
compiler.c, add after unary(), replace 1 line
\n\n

When the parser encounters a left parenthesis following an expression, it\ndispatches to a new parser function.

\n
compiler.c
\nadd after binary()
\n
static void call(bool canAssign) {\n  uint8_t argCount = argumentList();\n  emitBytes(OP_CALL, argCount);\n}\n
\n
compiler.c, add after binary()
\n\n

We’ve already consumed the ( token, so next we compile the arguments using a\nseparate argumentList() helper. That function returns the number of arguments\nit compiled. Each argument expression generates code that leaves its value on\nthe stack in preparation for the call. After that, we emit a new OP_CALL\ninstruction to invoke the function, using the argument count as an operand.

\n

We compile the arguments using this friend:

\n
compiler.c
\nadd after defineVariable()
\n
static uint8_t argumentList() {\n  uint8_t argCount = 0;\n  if (!check(TOKEN_RIGHT_PAREN)) {\n    do {\n      expression();\n      argCount++;\n    } while (match(TOKEN_COMMA));\n  }\n  consume(TOKEN_RIGHT_PAREN, "Expect ')' after arguments.");\n  return argCount;\n}\n
\n
compiler.c, add after defineVariable()
\n\n

That code should look familiar from jlox. We chew through arguments as long as\nwe find commas after each expression. Once we run out, we consume the final\nclosing parenthesis and we’re done.

\n

Well, almost. Back in jlox, we added a compile-time check that you don’t pass\nmore than 255 arguments to a call. At the time, I said that was because clox\nwould need a similar limit. Now you can see why—since we stuff the argument\ncount into the bytecode as a single-byte operand, we can only go up to 255. We\nneed to verify that in this compiler too.

\n
      expression();\n
compiler.c
\nin argumentList()
\n
      if (argCount == 255) {\n        error("Can't have more than 255 arguments.");\n      }\n
      argCount++;\n
\n
compiler.c, in argumentList()
\n\n

That’s the front end. Let’s skip over to the back end, with a quick stop in the\nmiddle to declare the new instruction.

\n
  OP_LOOP,\n
chunk.h
\nin enum OpCode
\n
  OP_CALL,\n
  OP_RETURN,\n
\n
chunk.h, in enum OpCode
\n\n

24 . 5 . 1Binding arguments to parameters

\n

Before we get to the implementation, we should think about what the stack looks\nlike at the point of a call and what we need to do from there. When we reach the\ncall instruction, we have already executed the expression for the function being\ncalled, followed by its arguments. Say our program looks like this:

\n
fun sum(a, b, c) {\n  return a + b + c;\n}\n\nprint 4 + sum(5, 6, 7);\n
\n

If we pause the VM right on the OP_CALL instruction for that call to sum(),\nthe stack looks like this:

\"Stack:\n

Picture this from the perspective of sum() itself. When the compiler compiled\nsum(), it automatically allocated slot zero. Then, after that, it allocated\nlocal slots for the parameters a, b, and c, in order. To perform a call to\nsum(), we need a CallFrame initialized with the function being called and a\nregion of stack slots that it can use. Then we need to collect the arguments\npassed to the function and get them into the corresponding slots for the\nparameters.

\n

When the VM starts executing the body of sum(), we want its stack window to\nlook like this:

\"The\n

Do you notice how the argument slots that the caller sets up and the parameter\nslots the callee needs are both in exactly the right order? How convenient! This\nis no coincidence. When I talked about each CallFrame having its own window into\nthe stack, I never said those windows must be disjoint. There’s nothing\npreventing us from overlapping them, like this:

\"The\n

The top of the caller’s stack contains the function\nbeing called followed by the arguments in order. We know the caller doesn’t have\nany other slots above those in use because any temporaries needed when\nevaluating argument expressions have been discarded by now. The bottom of the\ncallee’s stack overlaps so that the parameter slots exactly line up with where\nthe argument values already live.

\n\n

This means that we don’t need to do any work to “bind an argument to a\nparameter”. There’s no copying values between slots or across environments. The\narguments are already exactly where they need to be. It’s hard to beat that for\nperformance.

\n

Time to implement the call instruction.

\n
      }\n
vm.c
\nin run()
\n
      case OP_CALL: {\n        int argCount = READ_BYTE();\n        if (!callValue(peek(argCount), argCount)) {\n          return INTERPRET_RUNTIME_ERROR;\n        }\n        break;\n      }\n
      case OP_RETURN: {\n
\n
vm.c, in run()
\n\n

We need to know the function being called and the number of arguments passed to\nit. We get the latter from the instruction’s operand. That also tells us where\nto find the function on the stack by counting past the argument slots from the\ntop of the stack. We hand that data off to a separate callValue() function. If\nthat returns false, it means the call caused some sort of runtime error. When\nthat happens, we abort the interpreter.

\n

If callValue() is successful, there will be a new frame on the CallFrame stack\nfor the called function. The run() function has its own cached pointer to the\ncurrent frame, so we need to update that.

\n
          return INTERPRET_RUNTIME_ERROR;\n        }\n
vm.c
\nin run()
\n
        frame = &vm.frames[vm.frameCount - 1];\n
        break;\n
\n
vm.c, in run()
\n\n

Since the bytecode dispatch loop reads from that frame variable, when the VM\ngoes to execute the next instruction, it will read the ip from the newly\ncalled function’s CallFrame and jump to its code. The work for executing that\ncall begins here:

\n
vm.c
\nadd after peek()
\n
static bool callValue(Value callee, int argCount) {\n  if (IS_OBJ(callee)) {\n    switch (OBJ_TYPE(callee)) {\n      case OBJ_FUNCTION: \n        return call(AS_FUNCTION(callee), argCount);\n      default:\n        break; // Non-callable object type.\n    }\n  }\n  runtimeError("Can only call functions and classes.");\n  return false;\n}\n
\n
vm.c, add after peek()
\n\n\n

There’s more going on here than just initializing a new CallFrame. Because Lox\nis dynamically typed, there’s nothing to prevent a user from writing bad code\nlike:

\n
var notAFunction = 123;\nnotAFunction();\n
\n

If that happens, the runtime needs to safely report an error and halt. So the\nfirst thing we do is check the type of the value that we’re trying to call. If\nit’s not a function, we error out. Otherwise, the actual call happens here:

\n
vm.c
\nadd after peek()
\n
static bool call(ObjFunction* function, int argCount) {\n  CallFrame* frame = &vm.frames[vm.frameCount++];\n  frame->function = function;\n  frame->ip = function->chunk.code;\n  frame->slots = vm.stackTop - argCount - 1;\n  return true;\n}\n
\n
vm.c, add after peek()
\n\n

This simply initializes the next CallFrame on the stack. It stores a pointer to\nthe function being called and points the frame’s ip to the beginning of the\nfunction’s bytecode. Finally, it sets up the slots pointer to give the frame\nits window into the stack. The arithmetic there ensures that the arguments\nalready on the stack line up with the function’s parameters:

\"The\n

The funny little - 1 is to account for stack slot zero which the compiler set\naside for when we add methods later. The parameters start at slot one so we\nmake the window start one slot earlier to align them with the arguments.

\n

Before we move on, let’s add the new instruction to our disassembler.

\n
      return jumpInstruction("OP_LOOP", -1, chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_CALL:\n      return byteInstruction("OP_CALL", chunk, offset);\n
    case OP_RETURN:\n
\n
debug.c, in disassembleInstruction()
\n\n

And one more quick side trip. Now that we have a handy function for initiating a\nCallFrame, we may as well use it to set up the first frame for executing the\ntop-level code.

\n
  push(OBJ_VAL(function));\n
vm.c
\nin interpret()
\nreplace 4 lines
\n
  call(function, 0);\n
\n\n  return run();\n
\n
vm.c, in interpret(), replace 4 lines
\n\n

OK, now back to calls . . . 

\n

24 . 5 . 2Runtime error checking

\n

The overlapping stack windows work based on the assumption that a call passes\nexactly one argument for each of the function’s parameters. But, again, because\nLox ain’t statically typed, a foolish user could pass too many or too few\narguments. In Lox, we’ve defined that to be a runtime error, which we report\nlike so:

\n
static bool call(ObjFunction* function, int argCount) {\n
vm.c
\nin call()
\n
  if (argCount != function->arity) {\n    runtimeError("Expected %d arguments but got %d.",\n        function->arity, argCount);\n    return false;\n  }\n\n
  CallFrame* frame = &vm.frames[vm.frameCount++];\n
\n
vm.c, in call()
\n\n

Pretty straightforward. This is why we store the arity of each function inside\nthe ObjFunction for it.

\n

There’s another error we need to report that’s less to do with the user’s\nfoolishness than our own. Because the CallFrame array has a fixed size, we need\nto ensure a deep call chain doesn’t overflow it.

\n
  }\n\n
vm.c
\nin call()
\n
  if (vm.frameCount == FRAMES_MAX) {\n    runtimeError("Stack overflow.");\n    return false;\n  }\n\n
  CallFrame* frame = &vm.frames[vm.frameCount++];\n
\n
vm.c, in call()
\n\n

In practice, if a program gets anywhere close to this limit, there’s most likely\na bug in some runaway recursive code.

\n

24 . 5 . 3Printing stack traces

\n

While we’re on the subject of runtime errors, let’s spend a little time making\nthem more useful. Stopping on a runtime error is important to prevent the VM\nfrom crashing and burning in some ill-defined way. But simply aborting doesn’t\nhelp the user fix their code that caused that error.

\n

The classic tool to aid debugging runtime failures is a stack trace—a\nprint out of each function that was still executing when the program died, and\nwhere the execution was at the point that it died. Now that we have a call stack\nand we’ve conveniently stored each function’s name, we can show that entire\nstack when a runtime error disrupts the harmony of the user’s existence. It\nlooks like this:

\n
  fputs("\\n", stderr);\n\n
vm.c
\nin runtimeError()
\nreplace 4 lines
\n
  for (int i = vm.frameCount - 1; i >= 0; i--) {\n    CallFrame* frame = &vm.frames[i];\n    ObjFunction* function = frame->function;\n    size_t instruction = frame->ip - function->chunk.code - 1;\n    fprintf(stderr, "[line %d] in ", \n            function->chunk.lines[instruction]);\n    if (function->name == NULL) {\n      fprintf(stderr, "script\\n");\n    } else {\n      fprintf(stderr, "%s()\\n", function->name->chars);\n    }\n  }\n\n
  resetStack();\n}\n
\n
vm.c, in runtimeError(), replace 4 lines
\n\n\n

After printing the error message itself, we walk the call stack from top (the most recently called function) to bottom (the\ntop-level code). For each frame, we find the line number that corresponds to the\ncurrent ip inside that frame’s function. Then we print that line number along\nwith the function name.

\n\n

For example, if you run this broken program:

\n
fun a() { b(); }\nfun b() { c(); }\nfun c() {\n  c("too", "many");\n}\n\na();\n
\n

It prints out:

\n
Expected 0 arguments but got 2.\n[line 4] in c()\n[line 2] in b()\n[line 1] in a()\n[line 7] in script\n
\n

That doesn’t look too bad, does it?

\n

24 . 5 . 4Returning from functions

\n

We’re getting close. We can call functions, and the VM will execute them. But we\ncan’t return from them yet. We’ve had an OP_RETURN instruction for quite\nsome time, but it’s always had some kind of temporary code hanging out in it\njust to get us out of the bytecode loop. The time has arrived for a real\nimplementation.

\n
      case OP_RETURN: {\n
vm.c
\nin run()
\nreplace 2 lines
\n
        Value result = pop();\n        vm.frameCount--;\n        if (vm.frameCount == 0) {\n          pop();\n          return INTERPRET_OK;\n        }\n\n        vm.stackTop = frame->slots;\n        push(result);\n        frame = &vm.frames[vm.frameCount - 1];\n        break;\n
      }\n
\n
vm.c, in run(), replace 2 lines
\n\n

When a function returns a value, that value will be on top of the stack. We’re\nabout to discard the called function’s entire stack window, so we pop that\nreturn value off and hang on to it. Then we discard the CallFrame for the\nreturning function. If that was the very last CallFrame, it means we’ve finished\nexecuting the top-level code. The entire program is done, so we pop the main\nscript function from the stack and then exit the interpreter.

\n

Otherwise, we discard all of the slots the callee was using for its parameters\nand local variables. That includes the same slots the caller used to pass the\narguments. Now that the call is done, the caller doesn’t need them anymore. This\nmeans the top of the stack ends up right at the beginning of the returning\nfunction’s stack window.

\n

We push the return value back onto the stack at that new, lower location. Then\nwe update the run() function’s cached pointer to the current frame. Just like\nwhen we began a call, on the next iteration of the bytecode dispatch loop, the\nVM will read ip from that frame, and execution will jump back to the caller,\nright where it left off, immediately after the OP_CALL instruction.

\"Each\n

Note that we assume here that the function did actually return a value, but\na function can implicitly return by reaching the end of its body:

\n
fun noReturn() {\n  print "Do stuff";\n  // No return here.\n}\n\nprint noReturn(); // ???\n
\n

We need to handle that correctly too. The language is specified to implicitly\nreturn nil in that case. To make that happen, we add this:

\n
static void emitReturn() {\n
compiler.c
\nin emitReturn()
\n
  emitByte(OP_NIL);\n
  emitByte(OP_RETURN);\n}\n
\n
compiler.c, in emitReturn()
\n\n

The compiler calls emitReturn() to write the OP_RETURN instruction at the\nend of a function body. Now, before that, it emits an instruction to push nil\nonto the stack. And with that, we have working function calls! They can even\ntake parameters! It almost looks like we know what we’re doing here.

\n

24 . 6Return Statements

\n

If you want a function that returns something other than the implicit nil, you\nneed a return statement. Let’s get that working.

\n
    ifStatement();\n
compiler.c
\nin statement()
\n
  } else if (match(TOKEN_RETURN)) {\n    returnStatement();\n
  } else if (match(TOKEN_WHILE)) {\n
\n
compiler.c, in statement()
\n\n

When the compiler sees a return keyword, it goes here:

\n
compiler.c
\nadd after printStatement()
\n
static void returnStatement() {\n  if (match(TOKEN_SEMICOLON)) {\n    emitReturn();\n  } else {\n    expression();\n    consume(TOKEN_SEMICOLON, "Expect ';' after return value.");\n    emitByte(OP_RETURN);\n  }\n}\n
\n
compiler.c, add after printStatement()
\n\n

The return value expression is optional, so the parser looks for a semicolon\ntoken to tell if a value was provided. If there is no return value, the\nstatement implicitly returns nil. We implement that by calling emitReturn(),\nwhich emits an OP_NIL instruction. Otherwise, we compile the return value\nexpression and return it with an OP_RETURN instruction.

\n

This is the same OP_RETURN instruction we’ve already implemented—we don’t\nneed any new runtime code. This is quite a difference from jlox. There, we had\nto use exceptions to unwind the stack when a return statement was executed.\nThat was because you could return from deep inside some nested blocks. Since\njlox recursively walks the AST, that meant there were a bunch of Java method\ncalls we needed to escape out of.

\n

Our bytecode compiler flattens that all out. We do recursive descent during\nparsing, but at runtime, the VM’s bytecode dispatch loop is completely flat.\nThere is no recursion going on at the C level at all. So returning, even from\nwithin some nested blocks, is as straightforward as returning from the end of\nthe function’s body.

\n

We’re not totally done, though. The new return statement gives us a new\ncompile error to worry about. Returns are useful for returning from functions\nbut the top level of a Lox program is imperative code too. You shouldn’t be able\nto return from there.

\n
return "What?!";\n
\n\n

We’ve specified that it’s a compile error to have a return statement outside\nof any function, which we implement like so:

\n
static void returnStatement() {\n
compiler.c
\nin returnStatement()
\n
  if (current->type == TYPE_SCRIPT) {\n    error("Can't return from top-level code.");\n  }\n\n
  if (match(TOKEN_SEMICOLON)) {\n
\n
compiler.c, in returnStatement()
\n\n

This is one of the reasons we added that FunctionType enum to the compiler.

\n

24 . 7Native Functions

\n

Our VM is getting more powerful. We’ve got functions, calls, parameters,\nreturns. You can define lots of different functions that can call each other in\ninteresting ways. But, ultimately, they can’t really do anything. The only\nuser-visible thing a Lox program can do, regardless of its complexity, is print.\nTo add more capabilities, we need to expose them to the user.

\n

A programming language implementation reaches out and touches the material world\nthrough native functions. If you want to be able to write programs that\ncheck the time, read user input, or access the file system, we need to add\nnative functions—callable from Lox but implemented in C—that expose those\ncapabilities.

\n

At the language level, Lox is fairly complete—it’s got closures, classes,\ninheritance, and other fun stuff. One reason it feels like a toy language is\nbecause it has almost no native capabilities. We could turn it into a real\nlanguage by adding a long list of them.

\n

However, grinding through a pile of OS operations isn’t actually very\neducational. Once you’ve seen how to bind one piece of C code to Lox, you get\nthe idea. But you do need to see one, and even a single native function\nrequires us to build out all the machinery for interfacing Lox with C. So we’ll\ngo through that and do all the hard work. Then, when that’s done, we’ll add one\ntiny native function just to prove that it works.

\n

The reason we need new machinery is because, from the implementation’s\nperspective, native functions are different from Lox functions. When they are\ncalled, they don’t push a CallFrame, because there’s no bytecode code for that\nframe to point to. They have no bytecode chunk. Instead, they somehow reference\na piece of native C code.

\n

We handle this in clox by defining native functions as an entirely different\nobject type.

\n
} ObjFunction;\n
object.h
\nadd after struct ObjFunction
\n
\n\ntypedef Value (*NativeFn)(int argCount, Value* args);\n\ntypedef struct {\n  Obj obj;\n  NativeFn function;\n} ObjNative;\n
\n\nstruct ObjString {\n
\n
object.h, add after struct ObjFunction
\n\n

The representation is simpler than ObjFunction—merely an Obj header and a\npointer to the C function that implements the native behavior. The native\nfunction takes the argument count and a pointer to the first argument on the\nstack. It accesses the arguments through that pointer. Once it’s done, it\nreturns the result value.

\n

As always, a new object type carries some accoutrements with it. To create an\nObjNative, we declare a constructor-like function.

\n
ObjFunction* newFunction();\n
object.h
\nadd after newFunction()
\n
ObjNative* newNative(NativeFn function);\n
ObjString* takeString(char* chars, int length);\n
\n
object.h, add after newFunction()
\n\n

We implement that like so:

\n
object.c
\nadd after newFunction()
\n
ObjNative* newNative(NativeFn function) {\n  ObjNative* native = ALLOCATE_OBJ(ObjNative, OBJ_NATIVE);\n  native->function = function;\n  return native;\n}\n
\n
object.c, add after newFunction()
\n\n

The constructor takes a C function pointer to wrap in an ObjNative. It sets up\nthe object header and stores the function. For the header, we need a new object\ntype.

\n
typedef enum {\n  OBJ_FUNCTION,\n
object.h
\nin enum ObjType
\n
  OBJ_NATIVE,\n
  OBJ_STRING,\n} ObjType;\n
\n
object.h, in enum ObjType
\n\n

The VM also needs to know how to deallocate a native function object.

\n
    }\n
memory.c
\nin freeObject()
\n
    case OBJ_NATIVE:\n      FREE(ObjNative, object);\n      break;\n
    case OBJ_STRING: {\n
\n
memory.c, in freeObject()
\n\n

There isn’t much here since ObjNative doesn’t own any extra memory. The other\ncapability all Lox objects support is being printed.

\n
      break;\n
object.c
\nin printObject()
\n
    case OBJ_NATIVE:\n      printf("<native fn>");\n      break;\n
    case OBJ_STRING:\n
\n
object.c, in printObject()
\n\n

In order to support dynamic typing, we have a macro to see if a value is a\nnative function.

\n
#define IS_FUNCTION(value)     isObjType(value, OBJ_FUNCTION)\n
object.h
\n
#define IS_NATIVE(value)       isObjType(value, OBJ_NATIVE)\n
#define IS_STRING(value)       isObjType(value, OBJ_STRING)\n
\n
object.h
\n\n

Assuming that returns true, this macro extracts the C function pointer from a\nValue representing a native function:

\n
#define AS_FUNCTION(value)     ((ObjFunction*)AS_OBJ(value))\n
object.h
\n
#define AS_NATIVE(value) \\\n    (((ObjNative*)AS_OBJ(value))->function)\n
#define AS_STRING(value)       ((ObjString*)AS_OBJ(value))\n
\n
object.h
\n\n

All of this baggage lets the VM treat native functions like any other object.\nYou can store them in variables, pass them around, throw them birthday parties,\netc. Of course, the operation we actually care about is calling them—using\none as the left-hand operand in a call expression.

\n

Over in callValue() we add another type case.

\n
      case OBJ_FUNCTION: \n        return call(AS_FUNCTION(callee), argCount);\n
vm.c
\nin callValue()
\n
      case OBJ_NATIVE: {\n        NativeFn native = AS_NATIVE(callee);\n        Value result = native(argCount, vm.stackTop - argCount);\n        vm.stackTop -= argCount + 1;\n        push(result);\n        return true;\n      }\n
      default:\n
\n
vm.c, in callValue()
\n\n

If the object being called is a native function, we invoke the C function right\nthen and there. There’s no need to muck with CallFrames or anything. We just\nhand off to C, get the result, and stuff it back in the stack. This makes native\nfunctions as fast as we can get.

\n

With this, users should be able to call native functions, but there aren’t any\nto call. Without something like a foreign function interface, users can’t define\ntheir own native functions. That’s our job as VM implementers. We’ll start with\na helper to define a new native function exposed to Lox programs.

\n
vm.c
\nadd after runtimeError()
\n
static void defineNative(const char* name, NativeFn function) {\n  push(OBJ_VAL(copyString(name, (int)strlen(name))));\n  push(OBJ_VAL(newNative(function)));\n  tableSet(&vm.globals, AS_STRING(vm.stack[0]), vm.stack[1]);\n  pop();\n  pop();\n}\n
\n
vm.c, add after runtimeError()
\n\n

It takes a pointer to a C function and the name it will be known as in Lox.\nWe wrap the function in an ObjNative and then store that in a global variable\nwith the given name.

\n

You’re probably wondering why we push and pop the name and function on the\nstack. That looks weird, right? This is the kind of stuff you have to worry\nabout when garbage collection gets involved. Both\ncopyString() and newNative() dynamically allocate memory. That means once we\nhave a GC, they can potentially trigger a collection. If that happens, we need\nto ensure the collector knows we’re not done with the name and ObjFunction so\nthat it doesn’t free them out from under us. Storing them on the value stack\naccomplishes that.

\n\n

It feels silly, but after all of that work, we’re going to add only one\nlittle native function.

\n
vm.c
\nadd after variable vm
\n
static Value clockNative(int argCount, Value* args) {\n  return NUMBER_VAL((double)clock() / CLOCKS_PER_SEC);\n}\n
\n
vm.c, add after variable vm
\n\n

This returns the elapsed time since the program started running, in seconds. It’s\nhandy for benchmarking Lox programs. In Lox, we’ll name it clock().

\n
  initTable(&vm.strings);\n
vm.c
\nin initVM()
\n
\n\n  defineNative("clock", clockNative);\n
}\n
\n
vm.c, in initVM()
\n\n

To get to the C standard library clock() function, the “vm” module needs an\ninclude.

\n
#include <string.h>\n
vm.c
\n
#include <time.h>\n
\n\n#include "common.h"\n
\n
vm.c
\n\n

That was a lot of material to work through, but we did it! Type this in and try\nit out:

\n
fun fib(n) {\n  if (n < 2) return n;\n  return fib(n - 2) + fib(n - 1);\n}\n\nvar start = clock();\nprint fib(35);\nprint clock() - start;\n
\n

We can write a really inefficient recursive Fibonacci function. Even better, we\ncan measure just how inefficient it is. This is, of\ncourse, not the smartest way to calculate a Fibonacci number. But it is a good\nway to stress test a language implementation’s support for function calls. On my\nmachine, running this in clox is about five times faster than in jlox. That’s\nquite an improvement.

\n\n
\n

Challenges

\n
    \n
  1. \n

    Reading and writing the ip field is one of the most frequent operations\ninside the bytecode loop. Right now, we access it through a pointer to the\ncurrent CallFrame. That requires a pointer indirection which may force the\nCPU to bypass the cache and hit main memory. That can be a real performance\nsink.

    \n

    Ideally, we’d keep the ip in a native CPU register. C doesn’t let us\nrequire that without dropping into inline assembly, but we can structure\nthe code to encourage the compiler to make that optimization. If we store\nthe ip directly in a C local variable and mark it register, there’s a\ngood chance the C compiler will accede to our polite request.

    \n

    This does mean we need to be careful to load and store the local ip back\ninto the correct CallFrame when starting and ending function calls.\nImplement this optimization. Write a couple of benchmarks and see how it\naffects the performance. Do you think the extra code complexity is worth it?

    \n
    2. \n
  2. \n

    Native function calls are fast in part because we don’t validate that the\ncall passes as many arguments as the function expects. We really should, or\nan incorrect call to a native function without enough arguments could cause\nthe function to read uninitialized memory. Add arity checking.

    \n
    3. \n
  3. \n

    Right now, there’s no way for a native function to signal a runtime error.\nIn a real implementation, this is something we’d need to support because\nnative functions live in the statically typed world of C but are called\nfrom dynamically typed Lox land. If a user, say, tries to pass a string to\nsqrt(), that native function needs to report a runtime error.

    \n

    Extend the native function system to support that. How does this capability\naffect the performance of native calls?

    \n
    4. \n
  4. \n

    Add some more native functions to do things you find useful. Write some\nprograms using those. What did you add? How do they affect the feel of the\nlanguage and how practical it is?

    \n
    5. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/chunks-of-bytecode.html", "content": "\n\n\n\nChunks of Bytecode · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
14
\n

Chunks of Bytecode

\n\n
\n

If you find that you’re spending almost all your time on theory, start turning\nsome attention to practical things; it will improve your theories. If you find\nthat you’re spending almost all your time on practice, start turning some\nattention to theoretical things; it will improve your practice.

\n

Donald Knuth

\n
\n

We already have ourselves a complete implementation of Lox with jlox, so why\nisn’t the book over yet? Part of this is because jlox relies on the JVM to do lots of things for us. If we want to understand\nhow an interpreter works all the way down to the metal, we need to build those\nbits and pieces ourselves.

\n\n

An even more fundamental reason that jlox isn’t sufficient is that it’s too damn\nslow. A tree-walk interpreter is fine for some kinds of high-level, declarative\nlanguages. But for a general-purpose, imperative language—even a “scripting”\nlanguage like Lox—it won’t fly. Take this little script:

\n
fun fib(n) {\n  if (n < 2) return n;\n  return fib(n - 1) + fib(n - 2); \n}\n\nvar before = clock();\nprint fib(40);\nvar after = clock();\nprint after - before;\n
\n\n

On my laptop, that takes jlox about 72 seconds to execute. An equivalent C\nprogram finishes in half a second. Our dynamically typed scripting language is\nnever going to be as fast as a statically typed language with manual memory\nmanagement, but we don’t need to settle for more than two orders of magnitude\nslower.

\n

We could take jlox and run it in a profiler and start tuning and tweaking\nhotspots, but that will only get us so far. The execution model—walking the\nAST—is fundamentally the wrong design. We can’t micro-optimize that to the\nperformance we want any more than you can polish an AMC Gremlin into an SR-71\nBlackbird.

\n

We need to rethink the core model. This chapter introduces that model, bytecode,\nand begins our new interpreter, clox.

\n

14 . 1Bytecode?

\n

In engineering, few choices are without trade-offs. To best understand why we’re\ngoing with bytecode, let’s stack it up against a couple of alternatives.

\n

14 . 1 . 1Why not walk the AST?

\n

Our existing interpreter has a couple of things going for it:

\n
    \n
  • \n

    Well, first, we already wrote it. It’s done. And the main reason it’s done\nis because this style of interpreter is really simple to implement. The\nruntime representation of the code directly maps to the syntax. It’s\nvirtually effortless to get from the parser to the data structures we need\nat runtime.

    \n
    • \n
  • \n

    It’s portable. Our current interpreter is written in Java and runs on any\nplatform Java supports. We could write a new implementation in C using the\nsame approach and compile and run our language on basically every platform\nunder the sun.

    \n
    • \n
\n

Those are real advantages. But, on the other hand, it’s not memory-efficient.\nEach piece of syntax becomes an AST node. A tiny Lox expression like 1 + 2\nturns into a slew of objects with lots of pointers between them, something like:

\n

\n\"The\n

Each of those pointers adds an extra 32 or 64 bits of overhead to the object.\nWorse, sprinkling our data across the heap in a loosely connected web of objects\ndoes bad things for spatial locality.

\n\n

Modern CPUs process data way faster than they can pull it from RAM. To\ncompensate for that, chips have multiple layers of caching. If a piece of memory\nit needs is already in the cache, it can be loaded more quickly. We’re talking\nupwards of 100 times faster.

\n

How does data get into that cache? The machine speculatively stuffs things in\nthere for you. Its heuristic is pretty simple. Whenever the CPU reads a bit of\ndata from RAM, it pulls in a whole little bundle of adjacent bytes and stuffs\nthem in the cache.

\n

If our program next requests some data close enough to be inside that cache\nline, our CPU runs like a well-oiled conveyor belt in a factory. We really\nwant to take advantage of this. To use the cache effectively, the way we\nrepresent code in memory should be dense and ordered like it’s read.

\n

Now look up at that tree. Those sub-objects could be anywhere. Every step the tree-walker takes where it\nfollows a reference to a child node may step outside the bounds of the cache and\nforce the CPU to stall until a new lump of data can be slurped in from RAM. Just\nthe overhead of those tree nodes with all of their pointer fields and object\nheaders tends to push objects away from each other and out of the cache.

\n\n

Our AST walker has other overhead too around interface dispatch and the Visitor\npattern, but the locality issues alone are enough to justify a better code\nrepresentation.

\n

14 . 1 . 2Why not compile to native code?

\n

If you want to go real fast, you want to get all of those layers of\nindirection out of the way. Right down to the metal. Machine code. It even\nsounds fast. Machine code.

\n

Compiling directly to the native instruction set the chip supports is what the\nfastest languages do. Targeting native code has been the most efficient option\nsince way back in the early days when engineers actually handwrote programs in machine code.

\n\n

If you’ve never written any machine code, or its slightly more human-palatable\ncousin assembly code before, I’ll give you the gentlest of introductions. Native\ncode is a dense series of operations, encoded directly in binary. Each\ninstruction is between one and a few bytes long, and is almost mind-numbingly\nlow level. “Move a value from this address to this register.” “Add the integers\nin these two registers.” Stuff like that.

\n

The CPU cranks through the instructions, decoding and executing each one in\norder. There is no tree structure like our AST, and control flow is handled by\njumping from one point in the code directly to another. No indirection, no\noverhead, no unnecessary skipping around or chasing pointers.

\n

Lightning fast, but that performance comes at a cost. First of all, compiling to\nnative code ain’t easy. Most chips in wide use today have sprawling Byzantine\narchitectures with heaps of instructions that accreted over decades. They\nrequire sophisticated register allocation, pipelining, and instruction\nscheduling.

\n

And, of course, you’ve thrown portability out. Spend a\nfew years mastering some architecture and that still only gets you onto one of\nthe several popular instruction sets out there. To get your language on all of\nthem, you need to learn all of their instruction sets and write a separate back\nend for each one.

\n\n

14 . 1 . 3What is bytecode?

\n

Fix those two points in your mind. On one end, a tree-walk interpreter is\nsimple, portable, and slow. On the other, native code is complex and\nplatform-specific but fast. Bytecode sits in the middle. It retains the\nportability of a tree-walker—we won’t be getting our hands dirty with\nassembly code in this book. It sacrifices some simplicity to get a performance\nboost in return, though not as fast as going fully native.

\n

Structurally, bytecode resembles machine code. It’s a dense, linear sequence of\nbinary instructions. That keeps overhead low and plays nice with the cache.\nHowever, it’s a much simpler, higher-level instruction set than any real chip\nout there. (In many bytecode formats, each instruction is only a single byte\nlong, hence “bytecode”.)

\n

Imagine you’re writing a native compiler from some source language and you’re\ngiven carte blanche to define the easiest possible architecture to target.\nBytecode is kind of like that. It’s an idealized fantasy instruction set that\nmakes your life as the compiler writer easier.

\n

The problem with a fantasy architecture, of course, is that it doesn’t exist. We\nsolve that by writing an emulator—a simulated chip written in software that\ninterprets the bytecode one instruction at a time. A virtual machine (VM), if\nyou will.

\n

That emulation layer adds overhead, which is a key\nreason bytecode is slower than native code. But in return, it gives us\nportability. Write our VM in a language like C that is already supported on all\nthe machines we care about, and we can run our emulator on top of any hardware\nwe like.

\n\n

This is the path we’ll take with our new interpreter, clox. We’ll follow in the\nfootsteps of the main implementations of Python, Ruby, Lua, OCaml, Erlang, and\nothers. In many ways, our VM’s design will parallel the structure of our\nprevious interpreter:

\n

\"Phases

\n

Of course, we won’t implement the phases strictly in order. Like our previous\ninterpreter, we’ll bounce around, building up the implementation one language\nfeature at a time. In this chapter, we’ll get the skeleton of the application in\nplace and create the data structures needed to store and represent a chunk of\nbytecode.

\n

14 . 2Getting Started

\n

Where else to begin, but at main()? Fire up your\ntrusty text editor and start typing.

\n\n
main.c
\ncreate new file
\n
#include "common.h"\n\nint main(int argc, const char* argv[]) {\n  return 0;\n}\n
\n
main.c, create new file
\n\n

From this tiny seed, we will grow our entire VM. Since C provides us with so\nlittle, we first need to spend some time amending the soil. Some of that goes\ninto this header:

\n
common.h
\ncreate new file
\n
#ifndef clox_common_h\n#define clox_common_h\n\n#include <stdbool.h>\n#include <stddef.h>\n#include <stdint.h>\n\n#endif\n
\n
common.h, create new file
\n\n

There are a handful of types and constants we’ll use throughout the interpreter,\nand this is a convenient place to put them. For now, it’s the venerable NULL,\nsize_t, the nice C99 Boolean bool, and explicit-sized integer types—uint8_t and friends.

\n

14 . 3Chunks of Instructions

\n

Next, we need a module to define our code representation. I’ve been using\n“chunk” to refer to sequences of bytecode, so let’s make that the official name\nfor that module.

\n
chunk.h
\ncreate new file
\n
#ifndef clox_chunk_h\n#define clox_chunk_h\n\n#include "common.h"\n\n#endif\n
\n
chunk.h, create new file
\n\n

In our bytecode format, each instruction has a one-byte operation code\n(universally shortened to opcode). That number controls what kind of\ninstruction we’re dealing with—add, subtract, look up variable, etc. We\ndefine those here:

\n
#include "common.h"\n
chunk.h
\n
\n\ntypedef enum {\n  OP_RETURN,\n} OpCode;\n
\n\n#endif\n
\n
chunk.h
\n\n

For now, we start with a single instruction, OP_RETURN. When we have a\nfull-featured VM, this instruction will mean “return from the current function”.\nI admit this isn’t exactly useful yet, but we have to start somewhere, and this\nis a particularly simple instruction, for reasons we’ll get to later.

\n

14 . 3 . 1A dynamic array of instructions

\n

Bytecode is a series of instructions. Eventually, we’ll store some other data\nalong with the instructions, so let’s go ahead and create a struct to hold it\nall.

\n
} OpCode;\n
chunk.h
\nadd after enum OpCode
\n
\n\ntypedef struct {\n  uint8_t* code;\n} Chunk;\n
\n\n#endif\n
\n
chunk.h, add after enum OpCode
\n\n

At the moment, this is simply a wrapper around an array of bytes. Since we don’t\nknow how big the array needs to be before we start compiling a chunk, it must be\ndynamic. Dynamic arrays are one of my favorite data structures. That sounds like\nclaiming vanilla is my favorite ice cream flavor, but\nhear me out. Dynamic arrays provide:

\n\n
    \n
  • \n

    Cache-friendly, dense storage

    \n
    • \n
  • \n

    Constant-time indexed element lookup

    \n
    • \n
  • \n

    Constant-time appending to the end of the array

    \n
    • \n
\n

Those features are exactly why we used dynamic arrays all the time in jlox under\nthe guise of Java’s ArrayList class. Now that we’re in C, we get to roll our\nown. If you’re rusty on dynamic arrays, the idea is pretty simple. In addition\nto the array itself, we keep two numbers: the number of elements in the array we\nhave allocated (“capacity”) and how many of those allocated entries are actually\nin use (“count”).

\n
typedef struct {\n
chunk.h
\nin struct Chunk
\n
  int count;\n  int capacity;\n
  uint8_t* code;\n} Chunk;\n
\n
chunk.h, in struct Chunk
\n\n

When we add an element, if the count is less than the capacity, then there is\nalready available space in the array. We store the new element right in there\nand bump the count.

\n

\"Storing

\n

If we have no spare capacity, then the process is a little more involved.

\n

\"Growing

\n
    \n
  1. Allocate a new array with more capacity.
    2. \n
  2. Copy the existing elements from the old array to the new one.
    3. \n
  3. Store the new capacity.
    4. \n
  4. Delete the old array.
    5. \n
  5. Update code to point to the new array.
    6. \n
  6. Store the element in the new array now that there is room.
    7. \n
  7. Update the count.
    8. \n
\n\n

We have our struct ready, so let’s implement the functions to work with it. C\ndoesn’t have constructors, so we declare a function to initialize a new chunk.

\n
} Chunk;\n
chunk.h
\nadd after struct Chunk
\n
\n\nvoid initChunk(Chunk* chunk);\n
\n\n#endif\n
\n
chunk.h, add after struct Chunk
\n\n

And implement it thusly:

\n
chunk.c
\ncreate new file
\n
#include <stdlib.h>\n\n#include "chunk.h"\n\nvoid initChunk(Chunk* chunk) {\n  chunk->count = 0;\n  chunk->capacity = 0;\n  chunk->code = NULL;\n}\n
\n
chunk.c, create new file
\n\n

The dynamic array starts off completely empty. We don’t even allocate a raw\narray yet. To append a byte to the end of the chunk, we use a new function.

\n
void initChunk(Chunk* chunk);\n
chunk.h
\nadd after initChunk()
\n
void writeChunk(Chunk* chunk, uint8_t byte);\n
\n\n#endif\n
\n
chunk.h, add after initChunk()
\n\n

This is where the interesting work happens.

\n
chunk.c
\nadd after initChunk()
\n
void writeChunk(Chunk* chunk, uint8_t byte) {\n  if (chunk->capacity < chunk->count + 1) {\n    int oldCapacity = chunk->capacity;\n    chunk->capacity = GROW_CAPACITY(oldCapacity);\n    chunk->code = GROW_ARRAY(uint8_t, chunk->code,\n        oldCapacity, chunk->capacity);\n  }\n\n  chunk->code[chunk->count] = byte;\n  chunk->count++;\n}\n
\n
chunk.c, add after initChunk()
\n\n

The first thing we need to do is see if the current array already has capacity\nfor the new byte. If it doesn’t, then we first need to grow the array to make\nroom. (We also hit this case on the very first write when the array is NULL\nand capacity is 0.)

\n

To grow the array, first we figure out the new capacity and grow the array to\nthat size. Both of those lower-level memory operations are defined in a new\nmodule.

\n
#include "chunk.h"\n
chunk.c
\n
#include "memory.h"\n
\n\nvoid initChunk(Chunk* chunk) {\n
\n
chunk.c
\n\n

This is enough to get us started.

\n
memory.h
\ncreate new file
\n
#ifndef clox_memory_h\n#define clox_memory_h\n\n#include "common.h"\n\n#define GROW_CAPACITY(capacity) \\\n    ((capacity) < 8 ? 8 : (capacity) * 2)\n\n#endif\n
\n
memory.h, create new file
\n\n

This macro calculates a new capacity based on a given current capacity. In order\nto get the performance we want, the important part is that it scales based on\nthe old size. We grow by a factor of two, which is pretty typical. 1.5× is\nanother common choice.

\n

We also handle when the current capacity is zero. In that case, we jump straight\nto eight elements instead of starting at one. That avoids a little extra memory churn when the array is very\nsmall, at the expense of wasting a few bytes on very small chunks.

\n\n

Once we know the desired capacity, we create or grow the array to that size\nusing GROW_ARRAY().

\n
#define GROW_CAPACITY(capacity) \\\n    ((capacity) < 8 ? 8 : (capacity) * 2)\n
memory.h
\n
\n\n#define GROW_ARRAY(type, pointer, oldCount, newCount) \\\n    (type*)reallocate(pointer, sizeof(type) * (oldCount), \\\n        sizeof(type) * (newCount))\n\nvoid* reallocate(void* pointer, size_t oldSize, size_t newSize);\n
\n\n#endif\n
\n
memory.h
\n\n

This macro pretties up a function call to reallocate() where the real work\nhappens. The macro itself takes care of getting the size of the array’s element\ntype and casting the resulting void* back to a pointer of the right type.

\n

This reallocate() function is the single function we’ll use for all dynamic\nmemory management in clox—allocating memory, freeing it, and changing the\nsize of an existing allocation. Routing all of those operations through a single\nfunction will be important later when we add a garbage collector that needs to\nkeep track of how much memory is in use.

\n

The two size arguments passed to reallocate() control which operation to\nperform:

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
oldSizenewSizeOperation
0Non‑zeroAllocate new block.
Non‑zero0Free allocation.
Non‑zeroSmaller than oldSizeShrink existing allocation.
Non‑zeroLarger than oldSizeGrow existing allocation.
\n

That sounds like a lot of cases to handle, but here’s the implementation:

\n
memory.c
\ncreate new file
\n
#include <stdlib.h>\n\n#include "memory.h"\n\nvoid* reallocate(void* pointer, size_t oldSize, size_t newSize) {\n  if (newSize == 0) {\n    free(pointer);\n    return NULL;\n  }\n\n  void* result = realloc(pointer, newSize);\n  return result;\n}\n
\n
memory.c, create new file
\n\n

When newSize is zero, we handle the deallocation case ourselves by calling\nfree(). Otherwise, we rely on the C standard library’s realloc() function.\nThat function conveniently supports the other three aspects of our policy. When\noldSize is zero, realloc() is equivalent to calling malloc().

\n

The interesting cases are when both oldSize and newSize are not zero. Those\ntell realloc() to resize the previously allocated block. If the new size is\nsmaller than the existing block of memory, it simply updates the size of the block and returns the same pointer\nyou gave it. If the new size is larger, it attempts to grow the existing block\nof memory.

\n

It can do that only if the memory after that block isn’t already in use. If\nthere isn’t room to grow the block, realloc() instead allocates a new block\nof memory of the desired size, copies over the old bytes, frees the old block,\nand then returns a pointer to the new block. Remember, that’s exactly the\nbehavior we want for our dynamic array.

\n

Because computers are finite lumps of matter and not the perfect mathematical\nabstractions computer science theory would have us believe, allocation can fail\nif there isn’t enough memory and realloc() will return NULL. We should\nhandle that.

\n
  void* result = realloc(pointer, newSize);\n
memory.c
\nin reallocate()
\n
  if (result == NULL) exit(1);\n
  return result;\n
\n
memory.c, in reallocate()
\n\n

There’s not really anything useful that our VM can do if it can’t get the\nmemory it needs, but we at least detect that and abort the process immediately\ninstead of returning a NULL pointer and letting it go off the rails later.

\n\n

OK, we can create new chunks and write instructions to them. Are we done? Nope!\nWe’re in C now, remember, we have to manage memory ourselves, like in Ye Olden\nTimes, and that means freeing it too.

\n
void initChunk(Chunk* chunk);\n
chunk.h
\nadd after initChunk()
\n
void freeChunk(Chunk* chunk);\n
void writeChunk(Chunk* chunk, uint8_t byte);\n
\n
chunk.h, add after initChunk()
\n\n

The implementation is:

\n
chunk.c
\nadd after initChunk()
\n
void freeChunk(Chunk* chunk) {\n  FREE_ARRAY(uint8_t, chunk->code, chunk->capacity);\n  initChunk(chunk);\n}\n
\n
chunk.c, add after initChunk()
\n\n

We deallocate all of the memory and then call initChunk() to zero out the\nfields leaving the chunk in a well-defined empty state. To free the memory, we\nadd one more macro.

\n
#define GROW_ARRAY(type, pointer, oldCount, newCount) \\\n    (type*)reallocate(pointer, sizeof(type) * (oldCount), \\\n        sizeof(type) * (newCount))\n
memory.h
\n
\n\n#define FREE_ARRAY(type, pointer, oldCount) \\\n    reallocate(pointer, sizeof(type) * (oldCount), 0)\n
\n\nvoid* reallocate(void* pointer, size_t oldSize, size_t newSize);\n
\n
memory.h
\n\n

Like GROW_ARRAY(), this is a wrapper around a call to reallocate(). This one\nfrees the memory by passing in zero for the new size. I know, this is a lot of\nboring low-level stuff. Don’t worry, we’ll get a lot of use out of these in\nlater chapters and will get to program at a higher level. Before we can do that,\nthough, we gotta lay our own foundation.

\n

14 . 4Disassembling Chunks

\n

Now we have a little module for creating chunks of bytecode. Let’s try it out by\nhand-building a sample chunk.

\n
int main(int argc, const char* argv[]) {\n
main.c
\nin main()
\n
  Chunk chunk;\n  initChunk(&chunk);\n  writeChunk(&chunk, OP_RETURN);\n  freeChunk(&chunk);\n
  return 0;\n
\n
main.c, in main()
\n\n

Don’t forget the include.

\n
#include "common.h"\n
main.c
\n
#include "chunk.h"\n
\n\nint main(int argc, const char* argv[]) {\n
\n
main.c
\n\n

Run that and give it a try. Did it work? Uh . . . who knows? All we’ve done is push\nsome bytes around in memory. We have no human-friendly way to see what’s\nactually inside that chunk we made.

\n

To fix this, we’re going to create a disassembler. An assembler is an\nold-school program that takes a file containing human-readable mnemonic names\nfor CPU instructions like “ADD” and “MULT” and translates them to their binary\nmachine code equivalent. A disassembler goes in the other direction—given a\nblob of machine code, it spits out a textual listing of the instructions.

\n

We’ll implement something similar. Given a chunk, it\nwill print out all of the instructions in it. A Lox user won’t use this, but\nwe Lox maintainers will certainly benefit since it gives us a window into the\ninterpreter’s internal representation of code.

\n\n

In main(), after we create the chunk, we pass it to the disassembler.

\n
  initChunk(&chunk);\n  writeChunk(&chunk, OP_RETURN);\n
main.c
\nin main()
\n
\n\n  disassembleChunk(&chunk, "test chunk");\n
  freeChunk(&chunk);\n
\n
main.c, in main()
\n\n

Again, we whip up yet another module.

\n\n
#include "chunk.h"\n
main.c
\n
#include "debug.h"\n
\n\nint main(int argc, const char* argv[]) {\n
\n
main.c
\n\n

Here’s that header:

\n
debug.h
\ncreate new file
\n
#ifndef clox_debug_h\n#define clox_debug_h\n\n#include "chunk.h"\n\nvoid disassembleChunk(Chunk* chunk, const char* name);\nint disassembleInstruction(Chunk* chunk, int offset);\n\n#endif\n
\n
debug.h, create new file
\n\n

In main(), we call disassembleChunk() to disassemble all of the instructions\nin the entire chunk. That’s implemented in terms of the other function, which\njust disassembles a single instruction. It shows up here in the header because\nwe’ll call it from the VM in later chapters.

\n

Here’s a start at the implementation file:

\n
debug.c
\ncreate new file
\n
#include <stdio.h>\n\n#include "debug.h"\n\nvoid disassembleChunk(Chunk* chunk, const char* name) {\n  printf("== %s ==\\n", name);\n\n  for (int offset = 0; offset < chunk->count;) {\n    offset = disassembleInstruction(chunk, offset);\n  }\n}\n
\n
debug.c, create new file
\n\n

To disassemble a chunk, we print a little header (so we can tell which chunk\nwe’re looking at) and then crank through the bytecode, disassembling each\ninstruction. The way we iterate through the code is a little odd. Instead of\nincrementing offset in the loop, we let disassembleInstruction() do it for\nus. When we call that function, after disassembling the instruction at the given\noffset, it returns the offset of the next instruction. This is because, as\nwe’ll see later, instructions can have different sizes.

\n

The core of the “debug” module is this function:

\n
debug.c
\nadd after disassembleChunk()
\n
int disassembleInstruction(Chunk* chunk, int offset) {\n  printf("%04d ", offset);\n\n  uint8_t instruction = chunk->code[offset];\n  switch (instruction) {\n    case OP_RETURN:\n      return simpleInstruction("OP_RETURN", offset);\n    default:\n      printf("Unknown opcode %d\\n", instruction);\n      return offset + 1;\n  }\n}\n
\n
debug.c, add after disassembleChunk()
\n\n

First, it prints the byte offset of the given instruction—that tells us where\nin the chunk this instruction is. This will be a helpful signpost when we start\ndoing control flow and jumping around in the bytecode.

\n

Next, it reads a single byte from the bytecode at the given offset. That’s our\nopcode. We switch on that. For each kind of\ninstruction, we dispatch to a little utility function for displaying it. On the\noff chance that the given byte doesn’t look like an instruction at all—a bug\nin our compiler—we print that too. For the one instruction we do have,\nOP_RETURN, the display function is:

\n\n
debug.c
\nadd after disassembleChunk()
\n
static int simpleInstruction(const char* name, int offset) {\n  printf("%s\\n", name);\n  return offset + 1;\n}\n
\n
debug.c, add after disassembleChunk()
\n\n

There isn’t much to a return instruction, so all it does is print the name of\nthe opcode, then return the next byte offset past this instruction. Other\ninstructions will have more going on.

\n

If we run our nascent interpreter now, it actually prints something:

\n
== test chunk ==\n0000 OP_RETURN\n
\n

It worked! This is sort of the “Hello, world!” of our code representation. We\ncan create a chunk, write an instruction to it, and then extract that\ninstruction back out. Our encoding and decoding of the binary bytecode is\nworking.

\n

14 . 5Constants

\n

Now that we have a rudimentary chunk structure working, let’s start making it\nmore useful. We can store code in chunks, but what about data? Many values\nthe interpreter works with are created at runtime as the result of operations.

\n
1 + 2;\n
\n

The value 3 appears nowhere in the code here. However, the literals 1 and 2\ndo. To compile that statement to bytecode, we need some sort of instruction that\nmeans “produce a constant” and those literal values need to get stored in the\nchunk somewhere. In jlox, the Expr.Literal AST node held the value. We need a\ndifferent solution now that we don’t have a syntax tree.

\n

14 . 5 . 1Representing values

\n

We won’t be running any code in this chapter, but since constants have a foot\nin both the static and dynamic worlds of our interpreter, they force us to start\nthinking at least a little bit about how our VM should represent values.

\n

For now, we’re going to start as simple as possible—we’ll support only\ndouble-precision, floating-point numbers. This will obviously expand over time,\nso we’ll set up a new module to give ourselves room to grow.

\n
value.h
\ncreate new file
\n
#ifndef clox_value_h\n#define clox_value_h\n\n#include "common.h"\n\ntypedef double Value;\n\n#endif\n
\n
value.h, create new file
\n\n

This typedef abstracts how Lox values are concretely represented in C. That way,\nwe can change that representation without needing to go back and fix existing\ncode that passes around values.

\n

Back to the question of where to store constants in a chunk. For small\nfixed-size values like integers, many instruction sets store the value directly\nin the code stream right after the opcode. These are called immediate\ninstructions because the bits for the value are immediately after the opcode.

\n

That doesn’t work well for large or variable-sized constants like strings. In a\nnative compiler to machine code, those bigger constants get stored in a separate\n“constant data” region in the binary executable. Then, the instruction to load a\nconstant has an address or offset pointing to where the value is stored in that\nsection.

\n

Most virtual machines do something similar. For example, the Java Virtual\nMachine associates a constant pool with each compiled class.\nThat sounds good enough for clox to me. Each chunk will carry with it a list of\nthe values that appear as literals in the program. To keep things simpler, we’ll put all constants in there, even simple\nintegers.

\n\n

14 . 5 . 2Value arrays

\n

The constant pool is an array of values. The instruction to load a constant\nlooks up the value by index in that array. As with our bytecode array, the compiler doesn’t know how big the\narray needs to be ahead of time. So, again, we need a dynamic one. Since C\ndoesn’t have generic data structures, we’ll write another dynamic array data\nstructure, this time for Value.

\n\n
typedef double Value;\n
value.h
\n
\n\ntypedef struct {\n  int capacity;\n  int count;\n  Value* values;\n} ValueArray;\n
\n\n#endif\n
\n
value.h
\n\n

As with the bytecode array in Chunk, this struct wraps a pointer to an array\nalong with its allocated capacity and the number of elements in use. We also\nneed the same three functions to work with value arrays.

\n
} ValueArray;\n
value.h
\nadd after struct ValueArray
\n
\n\nvoid initValueArray(ValueArray* array);\nvoid writeValueArray(ValueArray* array, Value value);\nvoid freeValueArray(ValueArray* array);\n
\n\n#endif\n
\n
value.h, add after struct ValueArray
\n\n

The implementations will probably give you déjà vu. First, to create a new one:

\n
value.c
\ncreate new file
\n
#include <stdio.h>\n\n#include "memory.h"\n#include "value.h"\n\nvoid initValueArray(ValueArray* array) {\n  array->values = NULL;\n  array->capacity = 0;\n  array->count = 0;\n}\n
\n
value.c, create new file
\n\n

Once we have an initialized array, we can start adding\nvalues to it.

\n\n
value.c
\nadd after initValueArray()
\n
void writeValueArray(ValueArray* array, Value value) {\n  if (array->capacity < array->count + 1) {\n    int oldCapacity = array->capacity;\n    array->capacity = GROW_CAPACITY(oldCapacity);\n    array->values = GROW_ARRAY(Value, array->values,\n                               oldCapacity, array->capacity);\n  }\n\n  array->values[array->count] = value;\n  array->count++;\n}\n
\n
value.c, add after initValueArray()
\n\n

The memory-management macros we wrote earlier do let us reuse some of the logic\nfrom the code array, so this isn’t too bad. Finally, to release all memory used\nby the array:

\n
value.c
\nadd after writeValueArray()
\n
void freeValueArray(ValueArray* array) {\n  FREE_ARRAY(Value, array->values, array->capacity);\n  initValueArray(array);\n}\n
\n
value.c, add after writeValueArray()
\n\n

Now that we have growable arrays of values, we can add one to Chunk to store the\nchunk’s constants.

\n
  uint8_t* code;\n
chunk.h
\nin struct Chunk
\n
  ValueArray constants;\n
} Chunk;\n
\n
chunk.h, in struct Chunk
\n\n

Don’t forget the include.

\n
#include "common.h"\n
chunk.h
\n
#include "value.h"\n
\n\ntypedef enum {\n
\n
chunk.h
\n\n

Ah, C, and its Stone Age modularity story. Where were we? Right. When we\ninitialize a new chunk, we initialize its constant list too.

\n
  chunk->code = NULL;\n
chunk.c
\nin initChunk()
\n
  initValueArray(&chunk->constants);\n
}\n
\n
chunk.c, in initChunk()
\n\n

Likewise, we free the constants when we free the chunk.

\n
  FREE_ARRAY(uint8_t, chunk->code, chunk->capacity);\n
chunk.c
\nin freeChunk()
\n
  freeValueArray(&chunk->constants);\n
  initChunk(chunk);\n
\n
chunk.c, in freeChunk()
\n\n

Next, we define a convenience method to add a new constant to the chunk. Our\nyet-to-be-written compiler could write to the constant array inside Chunk\ndirectly—it’s not like C has private fields or anything—but it’s a little\nnicer to add an explicit function.

\n
void writeChunk(Chunk* chunk, uint8_t byte);\n
chunk.h
\nadd after writeChunk()
\n
int addConstant(Chunk* chunk, Value value);\n
\n\n#endif\n
\n
chunk.h, add after writeChunk()
\n\n

Then we implement it.

\n
chunk.c
\nadd after writeChunk()
\n
int addConstant(Chunk* chunk, Value value) {\n  writeValueArray(&chunk->constants, value);\n  return chunk->constants.count - 1;\n}\n
\n
chunk.c, add after writeChunk()
\n\n

After we add the constant, we return the index where the constant was appended\nso that we can locate that same constant later.

\n

14 . 5 . 3Constant instructions

\n

We can store constants in chunks, but we also need to execute them. In a\npiece of code like:

\n
print 1;\nprint 2;\n
\n

The compiled chunk needs to not only contain the values 1 and 2, but know when\nto produce them so that they are printed in the right order. Thus, we need an\ninstruction that produces a particular constant.

\n
typedef enum {\n
chunk.h
\nin enum OpCode
\n
  OP_CONSTANT,\n
  OP_RETURN,\n
\n
chunk.h, in enum OpCode
\n\n

When the VM executes a constant instruction, it “loads”\nthe constant for use. This new instruction is a little more complex than\nOP_RETURN. In the above example, we load two different constants. A single\nbare opcode isn’t enough to know which constant to load.

\n\n

To handle cases like this, our bytecode—like most others—allows\ninstructions to have operands. These are stored\nas binary data immediately after the opcode in the instruction stream and let us\nparameterize what the instruction does.

\n

\"OP_CONSTANT

\n

Each opcode determines how many operand bytes it has and what they mean. For\nexample, a simple operation like “return” may have no operands, where an\ninstruction for “load local variable” needs an operand to identify which\nvariable to load. Each time we add a new opcode to clox, we specify what its\noperands look like—its instruction format.

\n\n

In this case, OP_CONSTANT takes a single byte operand that specifies which\nconstant to load from the chunk’s constant array. Since we don’t have a compiler\nyet, we “hand-compile” an instruction in our test chunk.

\n
  initChunk(&chunk);\n
main.c
\nin main()
\n
\n\n  int constant = addConstant(&chunk, 1.2);\n  writeChunk(&chunk, OP_CONSTANT);\n  writeChunk(&chunk, constant);\n\n
  writeChunk(&chunk, OP_RETURN);\n
\n
main.c, in main()
\n\n

We add the constant value itself to the chunk’s constant pool. That returns the\nindex of the constant in the array. Then we write the constant instruction,\nstarting with its opcode. After that, we write the one-byte constant index\noperand. Note that writeChunk() can write opcodes or operands. It’s all raw\nbytes as far as that function is concerned.

\n

If we try to run this now, the disassembler is going to yell at us because it\ndoesn’t know how to decode the new instruction. Let’s fix that.

\n
  switch (instruction) {\n
debug.c
\nin disassembleInstruction()
\n
    case OP_CONSTANT:\n      return constantInstruction("OP_CONSTANT", chunk, offset);\n
    case OP_RETURN:\n
\n
debug.c, in disassembleInstruction()
\n\n

This instruction has a different instruction format, so we write a new helper\nfunction to disassemble it.

\n
debug.c
\nadd after disassembleChunk()
\n
static int constantInstruction(const char* name, Chunk* chunk,\n                               int offset) {\n  uint8_t constant = chunk->code[offset + 1];\n  printf("%-16s %4d '", name, constant);\n  printValue(chunk->constants.values[constant]);\n  printf("'\\n");\n}\n
\n
debug.c, add after disassembleChunk()
\n\n

There’s more going on here. As with OP_RETURN, we print out the name of the\nopcode. Then we pull out the constant index from the subsequent byte in the\nchunk. We print that index, but that isn’t super useful to us human readers. So\nwe also look up the actual constant value—since constants are known at\ncompile time after all—and display the value itself too.

\n

This requires some way to print a clox Value. That function will live in the\n“value” module, so we include that.

\n
#include "debug.h"\n
debug.c
\n
#include "value.h"\n
\n\nvoid disassembleChunk(Chunk* chunk, const char* name) {\n
\n
debug.c
\n\n

Over in that header, we declare:

\n
void freeValueArray(ValueArray* array);\n
value.h
\nadd after freeValueArray()
\n
void printValue(Value value);\n
\n\n#endif\n
\n
value.h, add after freeValueArray()
\n\n

And here’s an implementation:

\n
value.c
\nadd after freeValueArray()
\n
void printValue(Value value) {\n  printf("%g", value);\n}\n
\n
value.c, add after freeValueArray()
\n\n

Magnificent, right? As you can imagine, this is going to get more complex once\nwe add dynamic typing to Lox and have values of different types.

\n

Back in constantInstruction(), the only remaining piece is the return value.

\n
  printf("'\\n");\n
debug.c
\nin constantInstruction()
\n
  return offset + 2;\n
}\n
\n
debug.c, in constantInstruction()
\n\n

Remember that disassembleInstruction() also returns a number to tell the\ncaller the offset of the beginning of the next instruction. Where OP_RETURN\nwas only a single byte, OP_CONSTANT is two—one for the opcode and one for\nthe operand.

\n

14 . 6Line Information

\n

Chunks contain almost all of the information that the runtime needs from the\nuser’s source code. It’s kind of crazy to think that we can reduce all of the\ndifferent AST classes that we created in jlox down to an array of bytes and an\narray of constants. There’s only one piece of data we’re missing. We need it,\neven though the user hopes to never see it.

\n

When a runtime error occurs, we show the user the line number of the offending\nsource code. In jlox, those numbers live in tokens, which we in turn store in\nthe AST nodes. We need a different solution for clox now that we’ve ditched\nsyntax trees in favor of bytecode. Given any bytecode instruction, we need to be\nable to determine the line of the user’s source program that it was compiled\nfrom.

\n

There are a lot of clever ways we could encode this. I took the absolute simplest approach I could come up with, even though it’s\nembarrassingly inefficient with memory. In the chunk, we store a separate array\nof integers that parallels the bytecode. Each number in the array is the line\nnumber for the corresponding byte in the bytecode. When a runtime error occurs,\nwe look up the line number at the same index as the current instruction’s offset\nin the code array.

\n\n

To implement this, we add another array to Chunk.

\n
  uint8_t* code;\n
chunk.h
\nin struct Chunk
\n
  int* lines;\n
  ValueArray constants;\n
\n
chunk.h, in struct Chunk
\n\n

Since it exactly parallels the bytecode array, we don’t need a separate count or\ncapacity. Every time we touch the code array, we make a corresponding change to\nthe line number array, starting with initialization.

\n
  chunk->code = NULL;\n
chunk.c
\nin initChunk()
\n
  chunk->lines = NULL;\n
  initValueArray(&chunk->constants);\n
\n
chunk.c, in initChunk()
\n\n

And likewise deallocation:

\n
  FREE_ARRAY(uint8_t, chunk->code, chunk->capacity);\n
chunk.c
\nin freeChunk()
\n
  FREE_ARRAY(int, chunk->lines, chunk->capacity);\n
  freeValueArray(&chunk->constants);\n
\n
chunk.c, in freeChunk()
\n\n

When we write a byte of code to the chunk, we need to know what source line it\ncame from, so we add an extra parameter in the declaration of writeChunk().

\n
void freeChunk(Chunk* chunk);\n
chunk.h
\nfunction writeChunk()
\nreplace 1 line
\n
void writeChunk(Chunk* chunk, uint8_t byte, int line);\n
int addConstant(Chunk* chunk, Value value);\n
\n
chunk.h, function writeChunk(), replace 1 line
\n\n

And in the implementation:

\n
chunk.c
\nfunction writeChunk()
\nreplace 1 line
\n
void writeChunk(Chunk* chunk, uint8_t byte, int line) {\n
  if (chunk->capacity < chunk->count + 1) {\n
\n
chunk.c, function writeChunk(), replace 1 line
\n\n

When we allocate or grow the code array, we do the same for the line info too.

\n
    chunk->code = GROW_ARRAY(uint8_t, chunk->code,\n        oldCapacity, chunk->capacity);\n
chunk.c
\nin writeChunk()
\n
    chunk->lines = GROW_ARRAY(int, chunk->lines,\n        oldCapacity, chunk->capacity);\n
  }\n
\n
chunk.c, in writeChunk()
\n\n

Finally, we store the line number in the array.

\n
  chunk->code[chunk->count] = byte;\n
chunk.c
\nin writeChunk()
\n
  chunk->lines[chunk->count] = line;\n
  chunk->count++;\n
\n
chunk.c, in writeChunk()
\n\n

14 . 6 . 1Disassembling line information

\n

Alright, let’s try this out with our little, uh, artisanal chunk. First, since\nwe added a new parameter to writeChunk(), we need to fix those calls to pass\nin some—arbitrary at this point—line number.

\n
  int constant = addConstant(&chunk, 1.2);\n
main.c
\nin main()
\nreplace 4 lines
\n
  writeChunk(&chunk, OP_CONSTANT, 123);\n  writeChunk(&chunk, constant, 123);\n\n  writeChunk(&chunk, OP_RETURN, 123);\n
\n\n  disassembleChunk(&chunk, "test chunk");\n
\n
main.c, in main(), replace 4 lines
\n\n

Once we have a real front end, of course, the compiler will track the current\nline as it parses and pass that in.

\n

Now that we have line information for every instruction, let’s put it to good\nuse. In our disassembler, it’s helpful to show which source line each\ninstruction was compiled from. That gives us a way to map back to the original\ncode when we’re trying to figure out what some blob of bytecode is supposed to\ndo. After printing the offset of the instruction—the number of bytes from the\nbeginning of the chunk—we show its source line.

\n
int disassembleInstruction(Chunk* chunk, int offset) {\n  printf("%04d ", offset);\n
debug.c
\nin disassembleInstruction()
\n
  if (offset > 0 &&\n      chunk->lines[offset] == chunk->lines[offset - 1]) {\n    printf("   | ");\n  } else {\n    printf("%4d ", chunk->lines[offset]);\n  }\n
\n\n  uint8_t instruction = chunk->code[offset];\n
\n
debug.c, in disassembleInstruction()
\n\n

Bytecode instructions tend to be pretty fine-grained. A single line of source\ncode often compiles to a whole sequence of instructions. To make that more\nvisually clear, we show a | for any instruction that comes from the same\nsource line as the preceding one. The resulting output for our handwritten\nchunk looks like:

\n
== test chunk ==\n0000  123 OP_CONSTANT         0 '1.2'\n0002    | OP_RETURN\n
\n

We have a three-byte chunk. The first two bytes are a constant instruction that\nloads 1.2 from the chunk’s constant pool. The first byte is the OP_CONSTANT\nopcode and the second is the index in the constant pool. The third byte (at\noffset 2) is a single-byte return instruction.

\n

In the remaining chapters, we will flesh this out with lots more kinds of\ninstructions. But the basic structure is here, and we have everything we need\nnow to completely represent an executable piece of code at runtime in our\nvirtual machine. Remember that whole family of AST classes we defined in jlox?\nIn clox, we’ve reduced that down to three arrays: bytes of code, constant\nvalues, and line information for debugging.

\n

This reduction is a key reason why our new interpreter will be faster than jlox.\nYou can think of bytecode as a sort of compact serialization of the AST, highly\noptimized for how the interpreter will deserialize it in the order it needs as\nit executes. In the next chapter, we will see how the virtual machine does\nexactly that.

\n
\n

Challenges

\n
    \n
  1. \n

    Our encoding of line information is hilariously wasteful of memory. Given\nthat a series of instructions often correspond to the same source line, a\nnatural solution is something akin to run-length encoding of the line\nnumbers.

    \n

    Devise an encoding that compresses the line information for a\nseries of instructions on the same line. Change writeChunk() to write this\ncompressed form, and implement a getLine() function that, given the index\nof an instruction, determines the line where the instruction occurs.

    \n

    Hint: It’s not necessary for getLine() to be particularly efficient.\nSince it is called only when a runtime error occurs, it is well off the\ncritical path where performance matters.

    \n
    2. \n
  2. \n

    Because OP_CONSTANT uses only a single byte for its operand, a chunk may\nonly contain up to 256 different constants. That’s small enough that people\nwriting real-world code will hit that limit. We could use two or more bytes\nto store the operand, but that makes every constant instruction take up\nmore space. Most chunks won’t need that many unique constants, so that\nwastes space and sacrifices some locality in the common case to support the\nrare case.

    \n

    To balance those two competing aims, many instruction sets feature multiple\ninstructions that perform the same operation but with operands of different\nsizes. Leave our existing one-byte OP_CONSTANT instruction alone, and\ndefine a second OP_CONSTANT_LONG instruction. It stores the operand as a\n24-bit number, which should be plenty.

    \n

    Implement this function:

    \n
    void writeConstant(Chunk* chunk, Value value, int line) {\n  // Implement me...\n}\n
    \n

    It adds value to chunk’s constant array and then writes an appropriate\ninstruction to load the constant. Also add support to the disassembler for\nOP_CONSTANT_LONG instructions.

    \n

    Defining two instructions seems to be the best of both worlds. What\nsacrifices, if any, does it force on us?

    \n
    3. \n
  3. \n

    Our reallocate() function relies on the C standard library for dynamic\nmemory allocation and freeing. malloc() and free() aren’t magic. Find\na couple of open source implementations of them and explain how they work.\nHow do they keep track of which bytes are allocated and which are free?\nWhat is required to allocate a block of memory? Free it? How do they make\nthat efficient? What do they do about fragmentation?

    \n

    Hardcore mode: Implement reallocate() without calling realloc(),\nmalloc(), or free(). You are allowed to call malloc() once, at the\nbeginning of the interpreter’s execution, to allocate a single big block of\nmemory, which your reallocate() function has access to. It parcels out\nblobs of memory from that single region, your own personal heap. It’s your\njob to define how it does that.

    \n
    4. \n
\n
\n
\n

Design Note: Test Your Language

\n

We’re almost halfway through the book and one thing we haven’t talked about is\ntesting your language implementation. That’s not because testing isn’t\nimportant. I can’t possibly stress enough how vital it is to have a good,\ncomprehensive test suite for your language.

\n

I wrote a test suite for Lox (which you are welcome to use on your own\nLox implementation) before I wrote a single word of this book. Those tests found\ncountless bugs in my implementations.

\n

Tests are important in all software, but they’re even more important for a\nprogramming language for at least a couple of reasons:

\n
    \n
  • \n

    Users expect their programming languages to be rock solid. We are so\nused to mature, stable compilers and interpreters that “It’s your code, not\nthe compiler” is an ingrained part of software culture. If there\nare bugs in your language implementation, users will go through the full\nfive stages of grief before they can figure out what’s going on, and you\ndon’t want to put them through all that.

    \n
    • \n
  • \n

    A language implementation is a deeply interconnected piece of software.\nSome codebases are broad and shallow. If the file loading code is broken in\nyour text editor, it—hopefully!—won’t cause failures in the text\nrendering on screen. Language implementations are narrower and deeper,\nespecially the core of the interpreter that handles the language’s actual\nsemantics. That makes it easy for subtle bugs to creep in caused by weird\ninteractions between various parts of the system. It takes good tests to\nflush those out.

    \n
    • \n
  • \n

    The input to a language implementation is, by design, combinatorial.\nThere are an infinite number of possible programs a user could write, and\nyour implementation needs to run them all correctly. You obviously can’t\ntest that exhaustively, but you need to work hard to cover as much of the\ninput space as you can.

    \n
    • \n
  • \n

    Language implementations are often complex, constantly changing, and full\nof optimizations. That leads to gnarly code with lots of dark corners\nwhere bugs can hide.

    \n
    • \n
\n

All of that means you’re gonna want a lot of tests. But what tests? Projects\nI’ve seen focus mostly on end-to-end “language tests”. Each test is a program\nwritten in the language along with the output or errors it is expected to\nproduce. Then you have a test runner that pushes the test program through your\nlanguage implementation and validates that it does what it’s supposed to.\nWriting your tests in the language itself has a few nice advantages:

\n
    \n
  • \n

    The tests aren’t coupled to any particular API or internal architecture\ndecisions of the implementation. This frees you to reorganize or rewrite\nparts of your interpreter or compiler without needing to update a slew of\ntests.

    \n
    • \n
  • \n

    You can use the same tests for multiple implementations of the language.

    \n
    • \n
  • \n

    Tests can often be terse and easy to read and maintain since they are\nsimply scripts in your language.

    \n
    • \n
\n

It’s not all rosy, though:

\n
    \n
  • \n

    End-to-end tests help you determine if there is a bug, but not where the\nbug is. It can be harder to figure out where the erroneous code in the\nimplementation is because all the test tells you is that the right output\ndidn’t appear.

    \n
    • \n
  • \n

    It can be a chore to craft a valid program that tickles some obscure corner\nof the implementation. This is particularly true for highly optimized\ncompilers where you may need to write convoluted code to ensure that you\nend up on just the right optimization path where a bug may be hiding.

    \n
    • \n
  • \n

    The overhead can be high to fire up the interpreter, parse, compile, and\nrun each test script. With a big suite of tests—which you do want,\nremember—that can mean a lot of time spent waiting for the tests to\nfinish running.

    \n
    • \n
\n

I could go on, but I don’t want this to turn into a sermon. Also, I don’t\npretend to be an expert on how to test languages. I just want you to\ninternalize how important it is that you test yours. Seriously. Test your\nlanguage. You’ll thank me for it.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/classes-and-instances.html", "content": "\n\n\n\nClasses and Instances · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
27
\n

Classes and Instances

\n\n
\n

Caring too much for objects can destroy you. Only—if you care for a thing\nenough, it takes on a life of its own, doesn’t it? And isn’t the whole point\nof things—beautiful things—that they connect you to some larger beauty?

\n

Donna Tartt, The Goldfinch

\n
\n

The last area left to implement in clox is object-oriented programming. OOP is a bundle of intertwined features: classes, instances,\nfields, methods, initializers, and inheritance. Using relatively high-level\nJava, we packed all that into two chapters. Now that we’re coding in C, which\nfeels like building a model of the Eiffel tower out of toothpicks, we’ll devote\nthree chapters to covering the same territory. This makes for a leisurely stroll\nthrough the implementation. After strenuous chapters like closures and the\ngarbage collector, you have earned a rest. In fact, the book should be easy\nfrom here on out.

\n\n

In this chapter, we cover the first three features: classes, instances, and\nfields. This is the stateful side of object orientation. Then in the next two\nchapters, we will hang behavior and code reuse off of those objects.

\n

27 . 1Class Objects

\n

In a class-based object-oriented language, everything begins with classes. They\ndefine what sorts of objects exist in the program and are the factories used to\nproduce new instances. Going bottom-up, we’ll start with their runtime\nrepresentation and then hook that into the language.

\n

By this point, we’re well-acquainted with the process of adding a new object\ntype to the VM. We start with a struct.

\n
} ObjClosure;\n
object.h
\nadd after struct ObjClosure
\n
\n\ntypedef struct {\n  Obj obj;\n  ObjString* name;\n} ObjClass;\n
\n\nObjClosure* newClosure(ObjFunction* function);\n
\n
object.h, add after struct ObjClosure
\n\n

After the Obj header, we store the class’s name. This isn’t strictly needed for\nthe user’s program, but it lets us show the name at runtime for things like\nstack traces.

\n

The new type needs a corresponding case in the ObjType enum.

\n
typedef enum {\n
object.h
\nin enum ObjType
\n
  OBJ_CLASS,\n
  OBJ_CLOSURE,\n
\n
object.h, in enum ObjType
\n\n

And that type gets a corresponding pair of macros. First, for testing an\nobject’s type:

\n
#define OBJ_TYPE(value)        (AS_OBJ(value)->type)\n\n
object.h
\n
#define IS_CLASS(value)        isObjType(value, OBJ_CLASS)\n
#define IS_CLOSURE(value)      isObjType(value, OBJ_CLOSURE)\n
\n
object.h
\n\n

And then for casting a Value to an ObjClass pointer:

\n
#define IS_STRING(value)       isObjType(value, OBJ_STRING)\n\n
object.h
\n
#define AS_CLASS(value)        ((ObjClass*)AS_OBJ(value))\n
#define AS_CLOSURE(value)      ((ObjClosure*)AS_OBJ(value))\n
\n
object.h
\n\n

The VM creates new class objects using this function:

\n
} ObjClass;\n\n
object.h
\nadd after struct ObjClass
\n
ObjClass* newClass(ObjString* name);\n
ObjClosure* newClosure(ObjFunction* function);\n
\n
object.h, add after struct ObjClass
\n\n

The implementation lives over here:

\n
object.c
\nadd after allocateObject()
\n
ObjClass* newClass(ObjString* name) {\n  ObjClass* klass = ALLOCATE_OBJ(ObjClass, OBJ_CLASS);\n  klass->name = name; \n  return klass;\n}\n
\n
object.c, add after allocateObject()
\n\n

Pretty much all boilerplate. It takes in the class’s name as a string and stores\nit. Every time the user declares a new class, the VM will create a new one of\nthese ObjClass structs to represent it.

\n\n

When the VM no longer needs a class, it frees it like so:

\n
  switch (object->type) {\n
memory.c
\nin freeObject()
\n
    case OBJ_CLASS: {\n      FREE(ObjClass, object);\n      break;\n    } \n
    case OBJ_CLOSURE: {\n
\n
memory.c, in freeObject()
\n\n\n

We have a memory manager now, so we also need to support tracing through class\nobjects.

\n
  switch (object->type) {\n
memory.c
\nin blackenObject()
\n
    case OBJ_CLASS: {\n      ObjClass* klass = (ObjClass*)object;\n      markObject((Obj*)klass->name);\n      break;\n    }\n
    case OBJ_CLOSURE: {\n
\n
memory.c, in blackenObject()
\n\n

When the GC reaches a class object, it marks the class’s name to keep that\nstring alive too.

\n

The last operation the VM can perform on a class is printing it.

\n
  switch (OBJ_TYPE(value)) {\n
object.c
\nin printObject()
\n
    case OBJ_CLASS:\n      printf("%s", AS_CLASS(value)->name->chars);\n      break;\n
    case OBJ_CLOSURE:\n
\n
object.c, in printObject()
\n\n

A class simply says its own name.

\n

27 . 2Class Declarations

\n

Runtime representation in hand, we are ready to add support for classes to the\nlanguage. Next, we move into the parser.

\n
static void declaration() {\n
compiler.c
\nin declaration()
\nreplace 1 line
\n
  if (match(TOKEN_CLASS)) {\n    classDeclaration();\n  } else if (match(TOKEN_FUN)) {\n
    funDeclaration();\n
\n
compiler.c, in declaration(), replace 1 line
\n\n

Class declarations are statements, and the parser recognizes one by the leading\nclass keyword. The rest of the compilation happens over here:

\n
compiler.c
\nadd after function()
\n
static void classDeclaration() {\n  consume(TOKEN_IDENTIFIER, "Expect class name.");\n  uint8_t nameConstant = identifierConstant(&parser.previous);\n  declareVariable();\n\n  emitBytes(OP_CLASS, nameConstant);\n  defineVariable(nameConstant);\n\n  consume(TOKEN_LEFT_BRACE, "Expect '{' before class body.");\n  consume(TOKEN_RIGHT_BRACE, "Expect '}' after class body.");\n}\n
\n
compiler.c, add after function()
\n\n

Immediately after the class keyword is the class’s name. We take that\nidentifier and add it to the surrounding function’s constant table as a string.\nAs you just saw, printing a class shows its name, so the compiler needs to stuff\nthe name string somewhere that the runtime can find. The constant table is the\nway to do that.

\n

The class’s name is also used to bind the class\nobject to a variable of the same name. So we declare a variable with that\nidentifier right after consuming its token.

\n\n

Next, we emit a new instruction to actually create the class object at runtime.\nThat instruction takes the constant table index of the class’s name as an\noperand.

\n

After that, but before compiling the body of the class, we define the variable\nfor the class’s name. Declaring the variable adds it to the scope, but recall\nfrom a previous chapter that we can’t use the variable until it’s\ndefined. For classes, we define the variable before the body. That way, users\ncan refer to the containing class inside the bodies of its own methods. That’s\nuseful for things like factory methods that produce new instances of the class.

\n

Finally, we compile the body. We don’t have methods yet, so right now it’s\nsimply an empty pair of braces. Lox doesn’t require fields to be declared in the\nclass, so we’re done with the body—and the parser—for now.

\n

The compiler is emitting a new instruction, so let’s define that.

\n
  OP_RETURN,\n
chunk.h
\nin enum OpCode
\n
  OP_CLASS,\n
} OpCode;\n
\n
chunk.h, in enum OpCode
\n\n

And add it to the disassembler:

\n
    case OP_RETURN:\n      return simpleInstruction("OP_RETURN", offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_CLASS:\n      return constantInstruction("OP_CLASS", chunk, offset);\n
    default:\n
\n
debug.c, in disassembleInstruction()
\n\n

For such a large-seeming feature, the interpreter support is minimal.

\n
        break;\n      }\n
vm.c
\nin run()
\n
      case OP_CLASS:\n        push(OBJ_VAL(newClass(READ_STRING())));\n        break;\n
    }\n
\n
vm.c, in run()
\n\n

We load the string for the class’s name from the constant table and pass that to\nnewClass(). That creates a new class object with the given name. We push that\nonto the stack and we’re good. If the class is bound to a global variable, then\nthe compiler’s call to defineVariable() will emit code to store that object\nfrom the stack into the global variable table. Otherwise, it’s right where it\nneeds to be on the stack for a new local variable.

\n\n

There you have it, our VM supports classes now. You can run this:

\n
class Brioche {}\nprint Brioche;\n
\n

Unfortunately, printing is about all you can do with classes, so next is\nmaking them more useful.

\n

27 . 3Instances of Classes

\n

Classes serve two main purposes in a language:

\n
    \n
  • \n

    They are how you create new instances. Sometimes this involves a new\nkeyword, other times it’s a method call on the class object, but you usually\nmention the class by name somehow to get a new instance.

    \n
    • \n
  • \n

    They contain methods. These define how all instances of the class\nbehave.

    \n
    • \n
\n

We won’t get to methods until the next chapter, so for now we will only worry\nabout the first part. Before classes can create instances, we need a\nrepresentation for them.

\n
} ObjClass;\n
object.h
\nadd after struct ObjClass
\n
\n\ntypedef struct {\n  Obj obj;\n  ObjClass* klass;\n  Table fields; \n} ObjInstance;\n
\n\nObjClass* newClass(ObjString* name);\n
\n
object.h, add after struct ObjClass
\n\n

Instances know their class—each instance has a pointer to the class that it\nis an instance of. We won’t use this much in this chapter, but it will become\ncritical when we add methods.

\n

More important to this chapter is how instances store their state. Lox lets\nusers freely add fields to an instance at runtime. This means we need a storage\nmechanism that can grow. We could use a dynamic array, but we also want to look\nup fields by name as quickly as possible. There’s a data structure that’s just\nperfect for quickly accessing a set of values by name and—even more conveniently—we’ve already implemented it. Each instance stores\nits fields using a hash table.

\n\n

We only need to add an include, and we’ve got it.

\n
#include "chunk.h"\n
object.h
\n
#include "table.h"\n
#include "value.h"\n
\n
object.h
\n\n

This new struct gets a new object type.

\n
  OBJ_FUNCTION,\n
object.h
\nin enum ObjType
\n
  OBJ_INSTANCE,\n
  OBJ_NATIVE,\n
\n
object.h, in enum ObjType
\n\n

I want to slow down a bit here because the Lox language’s notion of “type” and\nthe VM implementation’s notion of “type” brush against each other in ways that\ncan be confusing. Inside the C code that makes clox, there are a number of\ndifferent types of Obj—ObjString, ObjClosure, etc. Each has its own internal\nrepresentation and semantics.

\n

In the Lox language, users can define their own classes—say Cake and Pie—and then create instances of those classes. From the user’s perspective, an\ninstance of Cake is a different type of object than an instance of Pie. But,\nfrom the VM’s perspective, every class the user defines is simply another value\nof type ObjClass. Likewise, each instance in the user’s program, no matter what\nclass it is an instance of, is an ObjInstance. That one VM object type covers\ninstances of all classes. The two worlds map to each other something like this:

\"A\n

Got it? OK, back to the implementation. We also get our usual macros.

\n
#define IS_FUNCTION(value)     isObjType(value, OBJ_FUNCTION)\n
object.h
\n
#define IS_INSTANCE(value)     isObjType(value, OBJ_INSTANCE)\n
#define IS_NATIVE(value)       isObjType(value, OBJ_NATIVE)\n
\n
object.h
\n\n

And:

\n
#define AS_FUNCTION(value)     ((ObjFunction*)AS_OBJ(value))\n
object.h
\n
#define AS_INSTANCE(value)     ((ObjInstance*)AS_OBJ(value))\n
#define AS_NATIVE(value) \\\n
\n
object.h
\n\n

Since fields are added after the instance is created, the “constructor” function\nonly needs to know the class.

\n
ObjFunction* newFunction();\n
object.h
\nadd after newFunction()
\n
ObjInstance* newInstance(ObjClass* klass);\n
ObjNative* newNative(NativeFn function);\n
\n
object.h, add after newFunction()
\n\n

We implement that function here:

\n
object.c
\nadd after newFunction()
\n
ObjInstance* newInstance(ObjClass* klass) {\n  ObjInstance* instance = ALLOCATE_OBJ(ObjInstance, OBJ_INSTANCE);\n  instance->klass = klass;\n  initTable(&instance->fields);\n  return instance;\n}\n
\n
object.c, add after newFunction()
\n\n

We store a reference to the instance’s class. Then we initialize the field\ntable to an empty hash table. A new baby object is born!

\n

At the sadder end of the instance’s lifespan, it gets freed.

\n
      FREE(ObjFunction, object);\n      break;\n    }\n
memory.c
\nin freeObject()
\n
    case OBJ_INSTANCE: {\n      ObjInstance* instance = (ObjInstance*)object;\n      freeTable(&instance->fields);\n      FREE(ObjInstance, object);\n      break;\n    }\n
    case OBJ_NATIVE:\n
\n
memory.c, in freeObject()
\n\n

The instance owns its field table so when freeing the instance, we also free the\ntable. We don’t explicitly free the entries in the table, because there may\nbe other references to those objects. The garbage collector will take care of\nthose for us. Here we free only the entry array of the table itself.

\n

Speaking of the garbage collector, it needs support for tracing through\ninstances.

\n
      markArray(&function->chunk.constants);\n      break;\n    }\n
memory.c
\nin blackenObject()
\n
    case OBJ_INSTANCE: {\n      ObjInstance* instance = (ObjInstance*)object;\n      markObject((Obj*)instance->klass);\n      markTable(&instance->fields);\n      break;\n    }\n
    case OBJ_UPVALUE:\n
\n
memory.c, in blackenObject()
\n\n

If the instance is alive, we need to keep its class around. Also, we need to\nkeep every object referenced by the instance’s fields. Most live objects that\nare not roots are reachable because some instance refers to the object in a\nfield. Fortunately, we already have a nice markTable() function to make\ntracing them easy.

\n

Less critical but still important is printing.

\n
      break;\n
object.c
\nin printObject()
\n
    case OBJ_INSTANCE:\n      printf("%s instance",\n             AS_INSTANCE(value)->klass->name->chars);\n      break;\n
    case OBJ_NATIVE:\n
\n
object.c, in printObject()
\n\n

An instance prints its name followed by “instance”.\n(The “instance” part is mainly so that classes and instances don’t print the\nsame.)

\n\n

The real fun happens over in the interpreter. Lox has no special new keyword.\nThe way to create an instance of a class is to invoke the class itself as if it\nwere a function. The runtime already supports function calls, and it checks the\ntype of object being called to make sure the user doesn’t try to invoke a number\nor other invalid type.

\n

We extend that runtime checking with a new case.

\n
    switch (OBJ_TYPE(callee)) {\n
vm.c
\nin callValue()
\n
      case OBJ_CLASS: {\n        ObjClass* klass = AS_CLASS(callee);\n        vm.stackTop[-argCount - 1] = OBJ_VAL(newInstance(klass));\n        return true;\n      }\n
      case OBJ_CLOSURE:\n
\n
vm.c, in callValue()
\n\n

If the value being called—the object that results when evaluating the\nexpression to the left of the opening parenthesis—is a class, then we treat\nit as a constructor call. We create a new instance of\nthe called class and store the result on the stack.

\n\n

We’re one step farther. Now we can define classes and create instances of them.

\n
class Brioche {}\nprint Brioche();\n
\n

Note the parentheses after Brioche on the second line now. This prints\n“Brioche instance”.

\n

27 . 4Get and Set Expressions

\n

Our object representation for instances can already store state, so all that\nremains is exposing that functionality to the user. Fields are accessed and\nmodified using get and set expressions. Not one to break with tradition, Lox\nuses the classic “dot” syntax:

\n
eclair.filling = "pastry creme";\nprint eclair.filling;\n
\n

The period—full stop for my English friends—works sort of like an infix operator. There is an expression to the\nleft that is evaluated first and produces an instance. After that is the .\nfollowed by a field name. Since there is a preceding operand, we hook this into\nthe parse table as an infix expression.

\n\n
  [TOKEN_COMMA]         = {NULL,     NULL,   PREC_NONE},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_DOT]           = {NULL,     dot,    PREC_CALL},\n
  [TOKEN_MINUS]         = {unary,    binary, PREC_TERM},\n
\n
compiler.c, replace 1 line
\n\n

As in other languages, the . operator binds tightly, with precedence as high\nas the parentheses in a function call. After the parser consumes the dot token,\nit dispatches to a new parse function.

\n
compiler.c
\nadd after call()
\n
static void dot(bool canAssign) {\n  consume(TOKEN_IDENTIFIER, "Expect property name after '.'.");\n  uint8_t name = identifierConstant(&parser.previous);\n\n  if (canAssign && match(TOKEN_EQUAL)) {\n    expression();\n    emitBytes(OP_SET_PROPERTY, name);\n  } else {\n    emitBytes(OP_GET_PROPERTY, name);\n  }\n}\n
\n
compiler.c, add after call()
\n\n

The parser expects to find a property name immediately\nafter the dot. We load that token’s lexeme into the constant table as a string\nso that the name is available at runtime.

\n\n

We have two new expression forms—getters and setters—that this one\nfunction handles. If we see an equals sign after the field name, it must be a\nset expression that is assigning to a field. But we don’t always allow an\nequals sign after the field to be compiled. Consider:

\n
a + b.c = 3\n
\n

This is syntactically invalid according to Lox’s grammar, which means our Lox\nimplementation is obligated to detect and report the error. If dot() silently\nparsed the = 3 part, we would incorrectly interpret the code as if the user\nhad written:

\n
a + (b.c = 3)\n
\n

The problem is that the = side of a set expression has much lower precedence\nthan the . part. The parser may call dot() in a context that is too high\nprecedence to permit a setter to appear. To avoid incorrectly allowing that, we\nparse and compile the equals part only when canAssign is true. If an equals\ntoken appears when canAssign is false, dot() leaves it alone and returns. In\nthat case, the compiler will eventually unwind up to parsePrecedence(), which\nstops at the unexpected = still sitting as the next token and reports an\nerror.

\n

If we find an = in a context where it is allowed, then we compile the\nexpression that follows. After that, we emit a new OP_SET_PROPERTY instruction. That takes a single operand for\nthe index of the property name in the constant table. If we didn’t compile a set\nexpression, we assume it’s a getter and emit an OP_GET_PROPERTY instruction,\nwhich also takes an operand for the property name.

\n\n

Now is a good time to define these two new instructions.

\n
  OP_SET_UPVALUE,\n
chunk.h
\nin enum OpCode
\n
  OP_GET_PROPERTY,\n  OP_SET_PROPERTY,\n
  OP_EQUAL,\n
\n
chunk.h, in enum OpCode
\n\n

And add support for disassembling them:

\n
      return byteInstruction("OP_SET_UPVALUE", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_GET_PROPERTY:\n      return constantInstruction("OP_GET_PROPERTY", chunk, offset);\n    case OP_SET_PROPERTY:\n      return constantInstruction("OP_SET_PROPERTY", chunk, offset);\n
    case OP_EQUAL:\n
\n
debug.c, in disassembleInstruction()
\n\n

27 . 4 . 1Interpreting getter and setter expressions

\n

Sliding over to the runtime, we’ll start with get expressions since those are a\nlittle simpler.

\n
      }\n
vm.c
\nin run()
\n
      case OP_GET_PROPERTY: {\n        ObjInstance* instance = AS_INSTANCE(peek(0));\n        ObjString* name = READ_STRING();\n\n        Value value;\n        if (tableGet(&instance->fields, name, &value)) {\n          pop(); // Instance.\n          push(value);\n          break;\n        }\n      }\n
      case OP_EQUAL: {\n
\n
vm.c, in run()
\n\n

When the interpreter reaches this instruction, the expression to the left of the\ndot has already been executed and the resulting instance is on top of the stack.\nWe read the field name from the constant pool and look it up in the instance’s\nfield table. If the hash table contains an entry with that name, we pop the\ninstance and push the entry’s value as the result.

\n

Of course, the field might not exist. In Lox, we’ve defined that to be a runtime\nerror. So we add a check for that and abort if it happens.

\n
          push(value);\n          break;\n        }\n
vm.c
\nin run()
\n
\n\n        runtimeError("Undefined property '%s'.", name->chars);\n        return INTERPRET_RUNTIME_ERROR;\n
      }\n      case OP_EQUAL: {\n
\n
vm.c, in run()
\n\n

There is another failure mode to handle which you’ve\nprobably noticed. The above code assumes the expression to the left of the dot\ndid evaluate to an ObjInstance. But there’s nothing preventing a user from\nwriting this:

\n
var obj = "not an instance";\nprint obj.field;\n
\n

The user’s program is wrong, but the VM still has to handle it with some grace.\nRight now, it will misinterpret the bits of the ObjString as an ObjInstance and,\nI don’t know, catch on fire or something definitely not graceful.

\n

In Lox, only instances are allowed to have fields. You can’t stuff a field onto\na string or number. So we need to check that the value is an instance before\naccessing any fields on it.

\n\n
      case OP_GET_PROPERTY: {\n
vm.c
\nin run()
\n
        if (!IS_INSTANCE(peek(0))) {\n          runtimeError("Only instances have properties.");\n          return INTERPRET_RUNTIME_ERROR;\n        }\n\n
        ObjInstance* instance = AS_INSTANCE(peek(0));\n
\n
vm.c, in run()
\n\n

If the value on the stack isn’t an instance, we report a runtime error and\nsafely exit.

\n

Of course, get expressions are not very useful when no instances have any\nfields. For that we need setters.

\n
        return INTERPRET_RUNTIME_ERROR;\n      }\n
vm.c
\nin run()
\n
      case OP_SET_PROPERTY: {\n        ObjInstance* instance = AS_INSTANCE(peek(1));\n        tableSet(&instance->fields, READ_STRING(), peek(0));\n        Value value = pop();\n        pop();\n        push(value);\n        break;\n      }\n
      case OP_EQUAL: {\n
\n
vm.c, in run()
\n\n

This is a little more complex than OP_GET_PROPERTY. When this executes, the\ntop of the stack has the instance whose field is being set and above that, the\nvalue to be stored. Like before, we read the instruction’s operand and find the\nfield name string. Using that, we store the value on top of the stack into the\ninstance’s field table.

\n

After that is a little stack juggling. We pop the\nstored value off, then pop the instance, and finally push the value back on. In\nother words, we remove the second element from the stack while leaving the top\nalone. A setter is itself an expression whose result is the assigned value, so\nwe need to leave that value on the stack. Here’s what I mean:

\n\n
class Toast {}\nvar toast = Toast();\nprint toast.jam = "grape"; // Prints "grape".\n
\n

Unlike when reading a field, we don’t need to worry about the hash table not\ncontaining the field. A setter implicitly creates the field if needed. We do\nneed to handle the user incorrectly trying to store a field on a value that\nisn’t an instance.

\n
      case OP_SET_PROPERTY: {\n
vm.c
\nin run()
\n
        if (!IS_INSTANCE(peek(1))) {\n          runtimeError("Only instances have fields.");\n          return INTERPRET_RUNTIME_ERROR;\n        }\n\n
        ObjInstance* instance = AS_INSTANCE(peek(1));\n
\n
vm.c, in run()
\n\n

Exactly like with get expressions, we check the value’s type and report a\nruntime error if it’s invalid. And, with that, the stateful side of Lox’s\nsupport for object-oriented programming is in place. Give it a try:

\n
class Pair {}\n\nvar pair = Pair();\npair.first = 1;\npair.second = 2;\nprint pair.first + pair.second; // 3.\n
\n

This doesn’t really feel very object-oriented. It’s more like a strange,\ndynamically typed variant of C where objects are loose struct-like bags of data.\nSort of a dynamic procedural language. But this is a big step in expressiveness.\nOur Lox implementation now lets users freely aggregate data into bigger units.\nIn the next chapter, we will breathe life into those inert blobs.

\n
\n

Challenges

\n
    \n
  1. \n

    Trying to access a non-existent field on an object immediately aborts the\nentire VM. The user has no way to recover from this runtime error, nor is\nthere any way to see if a field exists before trying to access it. It’s up\nto the user to ensure on their own that only valid fields are read.

    \n

    How do other dynamically typed languages handle missing fields? What do you\nthink Lox should do? Implement your solution.

    \n
    2. \n
  2. \n

    Fields are accessed at runtime by their string name. But that name must\nalways appear directly in the source code as an identifier token. A user\nprogram cannot imperatively build a string value and then use that as the\nname of a field. Do you think they should be able to? Devise a language\nfeature that enables that and implement it.

    \n
    3. \n
  3. \n

    Conversely, Lox offers no way to remove a field from an instance. You can\nset a field’s value to nil, but the entry in the hash table is still\nthere. How do other languages handle this? Choose and implement a strategy\nfor Lox.

    \n
    4. \n
  4. \n

    Because fields are accessed by name at runtime, working with instance state\nis slow. It’s technically a constant-time operation—thanks, hash tables—but the constant factors are relatively large. This is a major component\nof why dynamic languages are slower than statically typed ones.

    \n

    How do sophisticated implementations of dynamically typed languages cope\nwith and optimize this?

    \n
    5. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/classes.html", "content": "\n\n\n\nClasses · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
12
\n

Classes

\n\n
\n

One has no right to love or hate anything if one has not acquired a thorough\nknowledge of its nature. Great love springs from great knowledge of the\nbeloved object, and if you know it but little you will be able to love it only\na little or not at all.

\n

Leonardo da Vinci

\n
\n

We’re eleven chapters in, and the interpreter sitting on your machine is nearly\na complete scripting language. It could use a couple of built-in data structures\nlike lists and maps, and it certainly needs a core library for file I/O, user\ninput, etc. But the language itself is sufficient. We’ve got a little procedural\nlanguage in the same vein as BASIC, Tcl, Scheme (minus macros), and early\nversions of Python and Lua.

\n

If this were the ’80s, we’d stop here. But today, many popular languages support\n“object-oriented programming”. Adding that to Lox will give users a familiar set\nof tools for writing larger programs. Even if you personally don’t like OOP, this chapter and the next will help\nyou understand how others design and build object systems.

\n\n

12 . 1OOP and Classes

\n

There are three broad paths to object-oriented programming: classes,\nprototypes, and multimethods. Classes\ncame first and are the most popular style. With the rise of JavaScript (and to a\nlesser extent Lua), prototypes are more widely known than they used to be.\nI’ll talk more about those later. For Lox, we’re taking the, ahem, classic\napproach.

\n\n

Since you’ve written about a thousand lines of Java code with me already, I’m\nassuming you don’t need a detailed introduction to object orientation. The main\ngoal is to bundle data with the code that acts on it. Users do that by declaring\na class that:

\n

\n
    \n
  1. \n

    Exposes a constructor to create and initialize new instances of the\nclass

    \n
    2. \n
  2. \n

    Provides a way to store and access fields on instances

    \n
    3. \n
  3. \n

    Defines a set of methods shared by all instances of the class that\noperate on each instances’ state.

    \n
    4. \n
\n

That’s about as minimal as it gets. Most object-oriented languages, all the way\nback to Simula, also do inheritance to reuse behavior across classes. We’ll add\nthat in the next chapter. Even kicking that out, we still have a\nlot to get through. This is a big chapter and everything doesn’t quite come\ntogether until we have all of the above pieces, so gather your stamina.

\n\n

12 . 2Class Declarations

\n

Like we do, we’re gonna start with syntax. A class statement introduces a new\nname, so it lives in the declaration grammar rule.

\n
declaration    → classDecl\n               | funDecl\n               | varDecl\n               | statement ;\n\nclassDecl      → "class" IDENTIFIER "{" function* "}" ;\n
\n

The new classDecl rule relies on the function rule we defined\nearlier. To refresh your memory:

\n
function       → IDENTIFIER "(" parameters? ")" block ;\nparameters     → IDENTIFIER ( "," IDENTIFIER )* ;\n
\n

In plain English, a class declaration is the class keyword, followed by the\nclass’s name, then a curly-braced body. Inside that body is a list of method\ndeclarations. Unlike function declarations, methods don’t have a leading fun keyword. Each method is a name, parameter list, and\nbody. Here’s an example:

\n\n
class Breakfast {\n  cook() {\n    print "Eggs a-fryin'!";\n  }\n\n  serve(who) {\n    print "Enjoy your breakfast, " + who + ".";\n  }\n}\n
\n

Like most dynamically typed languages, fields are not explicitly listed in the\nclass declaration. Instances are loose bags of data and you can freely add\nfields to them as you see fit using normal imperative code.

\n

Over in our AST generator, the classDecl grammar rule gets its own statement\nnode.

\n
      "Block      : List<Stmt> statements",\n
tool/GenerateAst.java
\nin main()
\n
      "Class      : Token name, List<Stmt.Function> methods",\n
      "Expression : Expr expression",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

It stores the class’s name and the methods inside its body. Methods are\nrepresented by the existing Stmt.Function class that we use for function\ndeclaration AST nodes. That gives us all the bits of state that we need for a\nmethod: name, parameter list, and body.

\n

A class can appear anywhere a named declaration is allowed, triggered by the\nleading class keyword.

\n
    try {\n
lox/Parser.java
\nin declaration()
\n
      if (match(CLASS)) return classDeclaration();\n
      if (match(FUN)) return function("function");\n
\n
lox/Parser.java, in declaration()
\n\n

That calls out to:

\n
lox/Parser.java
\nadd after declaration()
\n
  private Stmt classDeclaration() {\n    Token name = consume(IDENTIFIER, "Expect class name.");\n    consume(LEFT_BRACE, "Expect '{' before class body.");\n\n    List<Stmt.Function> methods = new ArrayList<>();\n    while (!check(RIGHT_BRACE) && !isAtEnd()) {\n      methods.add(function("method"));\n    }\n\n    consume(RIGHT_BRACE, "Expect '}' after class body.");\n\n    return new Stmt.Class(name, methods);\n  }\n
\n
lox/Parser.java, add after declaration()
\n\n

There’s more meat to this than most of the other parsing methods, but it roughly\nfollows the grammar. We’ve already consumed the class keyword, so we look for\nthe expected class name next, followed by the opening curly brace. Once inside\nthe body, we keep parsing method declarations until we hit the closing brace.\nEach method declaration is parsed by a call to function(), which we defined\nback in the chapter where functions were introduced.

\n

Like we do in any open-ended loop in the parser, we also check for hitting the\nend of the file. That won’t happen in correct code since a class should have a\nclosing brace at the end, but it ensures the parser doesn’t get stuck in an\ninfinite loop if the user has a syntax error and forgets to correctly end the\nclass body.

\n

We wrap the name and list of methods into a Stmt.Class node and we’re done.\nPreviously, we would jump straight into the interpreter, but now we need to\nplumb the node through the resolver first.

\n
lox/Resolver.java
\nadd after visitBlockStmt()
\n
  @Override\n  public Void visitClassStmt(Stmt.Class stmt) {\n    declare(stmt.name);\n    define(stmt.name);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitBlockStmt()
\n\n

We aren’t going to worry about resolving the methods themselves yet, so for now\nall we need to do is declare the class using its name. It’s not common to\ndeclare a class as a local variable, but Lox permits it, so we need to handle it\ncorrectly.

\n

Now we interpret the class declaration.

\n
lox/Interpreter.java
\nadd after visitBlockStmt()
\n
  @Override\n  public Void visitClassStmt(Stmt.Class stmt) {\n    environment.define(stmt.name.lexeme, null);\n    LoxClass klass = new LoxClass(stmt.name.lexeme);\n    environment.assign(stmt.name, klass);\n    return null;\n  }\n
\n
lox/Interpreter.java, add after visitBlockStmt()
\n\n

This looks similar to how we execute function declarations. We declare the\nclass’s name in the current environment. Then we turn the class syntax node\ninto a LoxClass, the runtime representation of a class. We circle back and\nstore the class object in the variable we previously declared. That two-stage\nvariable binding process allows references to the class inside its own methods.

\n

We will refine it throughout the chapter, but the first draft of LoxClass looks\nlike this:

\n
lox/LoxClass.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.util.List;\nimport java.util.Map;\n\nclass LoxClass {\n  final String name;\n\n  LoxClass(String name) {\n    this.name = name;\n  }\n\n  @Override\n  public String toString() {\n    return name;\n  }\n}\n
\n
lox/LoxClass.java, create new file
\n\n

Literally a wrapper around a name. We don’t even store the methods yet. Not\nsuper useful, but it does have a toString() method so we can write a trivial\nscript and test that class objects are actually being parsed and executed.

\n
class DevonshireCream {\n  serveOn() {\n    return "Scones";\n  }\n}\n\nprint DevonshireCream; // Prints "DevonshireCream".\n
\n

12 . 3Creating Instances

\n

We have classes, but they don’t do anything yet. Lox doesn’t have “static”\nmethods that you can call right on the class itself, so without actual\ninstances, classes are useless. Thus instances are the next step.

\n

While some syntax and semantics are fairly standard across OOP languages, the\nway you create new instances isn’t. Ruby, following Smalltalk, creates instances\nby calling a method on the class object itself, a recursively graceful approach. Some, like C++ and Java,\nhave a new keyword dedicated to birthing a new object. Python has you “call”\nthe class itself like a function. (JavaScript, ever weird, sort of does both.)

\n\n

I took a minimal approach with Lox. We already have class objects, and we\nalready have function calls, so we’ll use call expressions on class objects to\ncreate new instances. It’s as if a class is a factory function that generates\ninstances of itself. This feels elegant to me, and also spares us the need to\nintroduce syntax like new. Therefore, we can skip past the front end straight\ninto the runtime.

\n

Right now, if you try this:

\n
class Bagel {}\nBagel();\n
\n

You get a runtime error. visitCallExpr() checks to see if the called object\nimplements LoxCallable and reports an error since LoxClass doesn’t. Not yet,\nthat is.

\n
import java.util.Map;\n\n
lox/LoxClass.java
\nreplace 1 line
\n
class LoxClass implements LoxCallable {\n
  final String name;\n
\n
lox/LoxClass.java, replace 1 line
\n\n

Implementing that interface requires two methods.

\n
lox/LoxClass.java
\nadd after toString()
\n
  @Override\n  public Object call(Interpreter interpreter,\n                     List<Object> arguments) {\n    LoxInstance instance = new LoxInstance(this);\n    return instance;\n  }\n\n  @Override\n  public int arity() {\n    return 0;\n  }\n
\n
lox/LoxClass.java, add after toString()
\n\n

The interesting one is call(). When you “call” a class, it instantiates a new\nLoxInstance for the called class and returns it. The arity() method is how the\ninterpreter validates that you passed the right number of arguments to a\ncallable. For now, we’ll say you can’t pass any. When we get to user-defined\nconstructors, we’ll revisit this.

\n

That leads us to LoxInstance, the runtime representation of an instance of a Lox\nclass. Again, our first implementation starts small.

\n
lox/LoxInstance.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.util.HashMap;\nimport java.util.Map;\n\nclass LoxInstance {\n  private LoxClass klass;\n\n  LoxInstance(LoxClass klass) {\n    this.klass = klass;\n  }\n\n  @Override\n  public String toString() {\n    return klass.name + " instance";\n  }\n}\n
\n
lox/LoxInstance.java, create new file
\n\n

Like LoxClass, it’s pretty bare bones, but we’re only getting started. If you\nwant to give it a try, here’s a script to run:

\n
class Bagel {}\nvar bagel = Bagel();\nprint bagel; // Prints "Bagel instance".\n
\n

This program doesn’t do much, but it’s starting to do something.

\n

12 . 4Properties on Instances

\n

We have instances, so we should make them useful. We’re at a fork in the road.\nWe could add behavior first—methods—or we could start with state—properties. We’re going to take the latter because, as we’ll see, the two get\nentangled in an interesting way and it will be easier to make sense of them if\nwe get properties working first.

\n

Lox follows JavaScript and Python in how it handles state. Every instance is an\nopen collection of named values. Methods on the instance’s class can access and\nmodify properties, but so can outside code.\nProperties are accessed using a . syntax.

\n\n
someObject.someProperty\n
\n

An expression followed by . and an identifier reads the property with that\nname from the object the expression evaluates to. That dot has the same\nprecedence as the parentheses in a function call expression, so we slot it into\nthe grammar by replacing the existing call rule with:

\n
call           → primary ( "(" arguments? ")" | "." IDENTIFIER )* ;\n
\n

After a primary expression, we allow a series of any mixture of parenthesized\ncalls and dotted property accesses. “Property access” is a mouthful, so from\nhere on out, we’ll call these “get expressions”.

\n

12 . 4 . 1Get expressions

\n

The syntax tree node is:

\n
      "Call     : Expr callee, Token paren, List<Expr> arguments",\n
tool/GenerateAst.java
\nin main()
\n
      "Get      : Expr object, Token name",\n
      "Grouping : Expr expression",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

Following the grammar, the new parsing code goes in our existing call()\nmethod.

\n
    while (true) { \n      if (match(LEFT_PAREN)) {\n        expr = finishCall(expr);\n
lox/Parser.java
\nin call()
\n
      } else if (match(DOT)) {\n        Token name = consume(IDENTIFIER,\n            "Expect property name after '.'.");\n        expr = new Expr.Get(expr, name);\n
      } else {\n        break;\n      }\n    }\n
\n
lox/Parser.java, in call()
\n\n

The outer while loop there corresponds to the * in the grammar rule. We zip\nalong the tokens building up a chain of calls and gets as we find parentheses\nand dots, like so:

\"Parsing\n

Instances of the new Expr.Get node feed into the resolver.

\n
lox/Resolver.java
\nadd after visitCallExpr()
\n
  @Override\n  public Void visitGetExpr(Expr.Get expr) {\n    resolve(expr.object);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitCallExpr()
\n\n

OK, not much to that. Since properties are looked up dynamically, they don’t get resolved. During resolution,\nwe recurse only into the expression to the left of the dot. The actual property\naccess happens in the interpreter.

\n\n
lox/Interpreter.java
\nadd after visitCallExpr()
\n
  @Override\n  public Object visitGetExpr(Expr.Get expr) {\n    Object object = evaluate(expr.object);\n    if (object instanceof LoxInstance) {\n      return ((LoxInstance) object).get(expr.name);\n    }\n\n    throw new RuntimeError(expr.name,\n        "Only instances have properties.");\n  }\n
\n
lox/Interpreter.java, add after visitCallExpr()
\n\n

First, we evaluate the expression whose property is being accessed. In Lox, only\ninstances of classes have properties. If the object is some other type like a\nnumber, invoking a getter on it is a runtime error.

\n

If the object is a LoxInstance, then we ask it to look up the property. It must\nbe time to give LoxInstance some actual state. A map will do fine.

\n
  private LoxClass klass;\n
lox/LoxInstance.java
\nin class LoxInstance
\n
  private final Map<String, Object> fields = new HashMap<>();\n
\n\n  LoxInstance(LoxClass klass) {\n
\n
lox/LoxInstance.java, in class LoxInstance
\n\n

Each key in the map is a property name and the corresponding value is the\nproperty’s value. To look up a property on an instance:

\n
lox/LoxInstance.java
\nadd after LoxInstance()
\n
  Object get(Token name) {\n    if (fields.containsKey(name.lexeme)) {\n      return fields.get(name.lexeme);\n    }\n\n    throw new RuntimeError(name, \n        "Undefined property '" + name.lexeme + "'.");\n  }\n
\n
lox/LoxInstance.java, add after LoxInstance()
\n\n\n

An interesting edge case we need to handle is what happens if the instance\ndoesn’t have a property with the given name. We could silently return some\ndummy value like nil, but my experience with languages like JavaScript is that\nthis behavior masks bugs more often than it does anything useful. Instead, we’ll\nmake it a runtime error.

\n

So the first thing we do is see if the instance actually has a field with the\ngiven name. Only then do we return it. Otherwise, we raise an error.

\n

Note how I switched from talking about “properties” to “fields”. There is a\nsubtle difference between the two. Fields are named bits of state stored\ndirectly in an instance. Properties are the named, uh, things, that a get\nexpression may return. Every field is a property, but as we’ll see later, not every property is a field.

\n\n

In theory, we can now read properties on objects. But since there’s no way to\nactually stuff any state into an instance, there are no fields to access. Before\nwe can test out reading, we must support writing.

\n

12 . 4 . 2Set expressions

\n

Setters use the same syntax as getters, except they appear on the left side of\nan assignment.

\n
someObject.someProperty = value;\n
\n

In grammar land, we extend the rule for assignment to allow dotted identifiers\non the left-hand side.

\n
assignment     → ( call "." )? IDENTIFIER "=" assignment\n               | logic_or ;\n
\n

Unlike getters, setters don’t chain. However, the reference to call allows any\nhigh-precedence expression before the last dot, including any number of\ngetters, as in:

\"breakfast.omelette.filling.meat\n

Note here that only the last part, the .meat is the setter. The\n.omelette and .filling parts are both get expressions.

\n

Just as we have two separate AST nodes for variable access and variable\nassignment, we need a second setter node to\ncomplement our getter node.

\n
      "Logical  : Expr left, Token operator, Expr right",\n
tool/GenerateAst.java
\nin main()
\n
      "Set      : Expr object, Token name, Expr value",\n
      "Unary    : Token operator, Expr right",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

In case you don’t remember, the way we handle assignment in the parser is a\nlittle funny. We can’t easily tell that a series of tokens is the left-hand side\nof an assignment until we reach the =. Now that our assignment grammar rule\nhas call on the left side, which can expand to arbitrarily large expressions,\nthat final = may be many tokens away from the point where we need to know\nwe’re parsing an assignment.

\n

Instead, the trick we do is parse the left-hand side as a normal expression.\nThen, when we stumble onto the equal sign after it, we take the expression we\nalready parsed and transform it into the correct syntax tree node for the\nassignment.

\n

We add another clause to that transformation to handle turning an Expr.Get\nexpression on the left into the corresponding Expr.Set.

\n
        return new Expr.Assign(name, value);\n
lox/Parser.java
\nin assignment()
\n
      } else if (expr instanceof Expr.Get) {\n        Expr.Get get = (Expr.Get)expr;\n        return new Expr.Set(get.object, get.name, value);\n
      }\n
\n
lox/Parser.java, in assignment()
\n\n

That’s parsing our syntax. We push that node through into the resolver.

\n
lox/Resolver.java
\nadd after visitLogicalExpr()
\n
  @Override\n  public Void visitSetExpr(Expr.Set expr) {\n    resolve(expr.value);\n    resolve(expr.object);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitLogicalExpr()
\n\n

Again, like Expr.Get, the property itself is dynamically evaluated, so there’s\nnothing to resolve there. All we need to do is recurse into the two\nsubexpressions of Expr.Set, the object whose property is being set, and the\nvalue it’s being set to.

\n

That leads us to the interpreter.

\n
lox/Interpreter.java
\nadd after visitLogicalExpr()
\n
  @Override\n  public Object visitSetExpr(Expr.Set expr) {\n    Object object = evaluate(expr.object);\n\n    if (!(object instanceof LoxInstance)) { \n      throw new RuntimeError(expr.name,\n                             "Only instances have fields.");\n    }\n\n    Object value = evaluate(expr.value);\n    ((LoxInstance)object).set(expr.name, value);\n    return value;\n  }\n
\n
lox/Interpreter.java, add after visitLogicalExpr()
\n\n

We evaluate the object whose property is being set and check to see if it’s a\nLoxInstance. If not, that’s a runtime error. Otherwise, we evaluate the value\nbeing set and store it on the instance. That relies on a new method in\nLoxInstance.

\n\n
lox/LoxInstance.java
\nadd after get()
\n
  void set(Token name, Object value) {\n    fields.put(name.lexeme, value);\n  }\n
\n
lox/LoxInstance.java, add after get()
\n\n

No real magic here. We stuff the values straight into the Java map where fields\nlive. Since Lox allows freely creating new fields on instances, there’s no need\nto see if the key is already present.

\n

12 . 5Methods on Classes

\n

You can create instances of classes and stuff data into them, but the class\nitself doesn’t really do anything. Instances are just maps and all instances\nare more or less the same. To make them feel like instances of classes, we\nneed behavior—methods.

\n

Our helpful parser already parses method declarations, so we’re good there. We\nalso don’t need to add any new parser support for method calls. We already\nhave . (getters) and () (function calls). A “method call” simply chains\nthose together.

\"The\n

That raises an interesting question. What happens when those two expressions are\npulled apart? Assuming that method in this example is a method on the class of\nobject and not a field on the instance, what should the following piece of\ncode do?

\n
var m = object.method;\nm(argument);\n
\n

This program “looks up” the method and stores the result—whatever that is—in a variable and then calls that object later. Is this allowed? Can you treat a\nmethod like it’s a function on the instance?

\n

What about the other direction?

\n
class Box {}\n\nfun notMethod(argument) {\n  print "called function with " + argument;\n}\n\nvar box = Box();\nbox.function = notMethod;\nbox.function("argument");\n
\n

This program creates an instance and then stores a function in a field on it.\nThen it calls that function using the same syntax as a method call. Does that\nwork?

\n

Different languages have different answers to these questions. One could write a\ntreatise on it. For Lox, we’ll say the answer to both of these is yes, it does\nwork. We have a couple of reasons to justify that. For the second example—calling a function stored in a field—we want to support that because\nfirst-class functions are useful and storing them in fields is a perfectly\nnormal thing to do.

\n

The first example is more obscure. One motivation is that users generally expect\nto be able to hoist a subexpression out into a local variable without changing\nthe meaning of the program. You can take this:

\n
breakfast(omelette.filledWith(cheese), sausage);\n
\n

And turn it into this:

\n
var eggs = omelette.filledWith(cheese);\nbreakfast(eggs, sausage);\n
\n

And it does the same thing. Likewise, since the . and the () in a method\ncall are two separate expressions, it seems you should be able to hoist the\nlookup part into a variable and then call it later. We need to think carefully about what the thing\nyou get when you look up a method is, and how it behaves, even in weird cases\nlike:

\n\n
class Person {\n  sayName() {\n    print this.name;\n  }\n}\n\nvar jane = Person();\njane.name = "Jane";\n\nvar method = jane.sayName;\nmethod(); // ?\n
\n

If you grab a handle to a method on some instance and call it later, does it\n“remember” the instance it was pulled off from? Does this inside the method\nstill refer to that original object?

\n

Here’s a more pathological example to bend your brain:

\n
class Person {\n  sayName() {\n    print this.name;\n  }\n}\n\nvar jane = Person();\njane.name = "Jane";\n\nvar bill = Person();\nbill.name = "Bill";\n\nbill.sayName = jane.sayName;\nbill.sayName(); // ?\n
\n

Does that last line print “Bill” because that’s the instance that we called\nthe method through, or “Jane” because it’s the instance where we first grabbed\nthe method?

\n

Equivalent code in Lua and JavaScript would print “Bill”. Those languages don’t\nreally have a notion of “methods”. Everything is sort of functions-in-fields, so\nit’s not clear that jane “owns” sayName any more than bill does.

\n

Lox, though, has real class syntax so we do know which callable things are\nmethods and which are functions. Thus, like Python, C#, and others, we will have\nmethods “bind” this to the original instance when the method is first grabbed.\nPython calls these bound methods.

\n\n

In practice, that’s usually what you want. If you take a reference to a method\non some object so you can use it as a callback later, you want to remember the\ninstance it belonged to, even if that callback happens to be stored in a field\non some other object.

\n

OK, that’s a lot of semantics to load into your head. Forget about the edge\ncases for a bit. We’ll get back to those. For now, let’s get basic method calls\nworking. We’re already parsing the method declarations inside the class body, so\nthe next step is to resolve them.

\n
    define(stmt.name);\n
lox/Resolver.java
\nin visitClassStmt()
\n
\n\n    for (Stmt.Function method : stmt.methods) {\n      FunctionType declaration = FunctionType.METHOD;\n      resolveFunction(method, declaration); \n    }\n\n
    return null;\n
\n
lox/Resolver.java, in visitClassStmt()
\n\n\n

We iterate through the methods in the class body and call the\nresolveFunction() method we wrote for handling function declarations already.\nThe only difference is that we pass in a new FunctionType enum value.

\n
    NONE,\n
    FUNCTION,\n
lox/Resolver.java
\nin enum FunctionType
\nadd “,” to previous line
\n
    METHOD\n
  }\n
\n
lox/Resolver.java, in enum FunctionType, add “,” to previous line
\n\n

That’s going to be important when we resolve this expressions. For now, don’t\nworry about it. The interesting stuff is in the interpreter.

\n
    environment.define(stmt.name.lexeme, null);\n
lox/Interpreter.java
\nin visitClassStmt()
\nreplace 1 line
\n
\n\n    Map<String, LoxFunction> methods = new HashMap<>();\n    for (Stmt.Function method : stmt.methods) {\n      LoxFunction function = new LoxFunction(method, environment);\n      methods.put(method.name.lexeme, function);\n    }\n\n    LoxClass klass = new LoxClass(stmt.name.lexeme, methods);\n
    environment.assign(stmt.name, klass);\n
\n
lox/Interpreter.java, in visitClassStmt(), replace 1 line
\n\n

When we interpret a class declaration statement, we turn the syntactic\nrepresentation of the class—its AST node—into its runtime representation.\nNow, we need to do that for the methods contained in the class as well. Each\nmethod declaration blossoms into a LoxFunction object.

\n

We take all of those and wrap them up into a map, keyed by the method names.\nThat gets stored in LoxClass.

\n
  final String name;\n
lox/LoxClass.java
\nin class LoxClass
\nreplace 4 lines
\n
  private final Map<String, LoxFunction> methods;\n\n  LoxClass(String name, Map<String, LoxFunction> methods) {\n    this.name = name;\n    this.methods = methods;\n  }\n
\n\n  @Override\n  public String toString() {\n
\n
lox/LoxClass.java, in class LoxClass, replace 4 lines
\n\n

Where an instance stores state, the class stores behavior. LoxInstance has its\nmap of fields, and LoxClass gets a map of methods. Even though methods are\nowned by the class, they are still accessed through instances of that class.

\n
  Object get(Token name) {\n    if (fields.containsKey(name.lexeme)) {\n      return fields.get(name.lexeme);\n    }\n\n
lox/LoxInstance.java
\nin get()
\n
    LoxFunction method = klass.findMethod(name.lexeme);\n    if (method != null) return method;\n\n
    throw new RuntimeError(name, \n        "Undefined property '" + name.lexeme + "'.");\n
\n
lox/LoxInstance.java, in get()
\n\n

When looking up a property on an instance, if we don’t find a matching field, we look for a method with that name\non the instance’s class. If found, we return that. This is where the distinction\nbetween “field” and “property” becomes meaningful. When accessing a property,\nyou might get a field—a bit of state stored on the instance—or you could\nhit a method defined on the instance’s class.

\n

The method is looked up using this:

\n\n
lox/LoxClass.java
\nadd after LoxClass()
\n
  LoxFunction findMethod(String name) {\n    if (methods.containsKey(name)) {\n      return methods.get(name);\n    }\n\n    return null;\n  }\n
\n
lox/LoxClass.java, add after LoxClass()
\n\n

You can probably guess this method is going to get more interesting later. For\nnow, a simple map lookup on the class’s method table is enough to get us\nstarted. Give it a try:

\n

\n
class Bacon {\n  eat() {\n    print "Crunch crunch crunch!";\n  }\n}\n\nBacon().eat(); // Prints "Crunch crunch crunch!".\n
\n\n

12 . 6This

\n

We can define both behavior and state on objects, but they aren’t tied together\nyet. Inside a method, we have no way to access the fields of the “current”\nobject—the instance that the method was called on—nor can we call other\nmethods on that same object.

\n

To get at that instance, it needs a name. Smalltalk,\nRuby, and Swift use “self”. Simula, C++, Java, and others use “this”. Python\nuses “self” by convention, but you can technically call it whatever you like.

\n\n

For Lox, since we generally hew to Java-ish style, we’ll go with “this”. Inside\na method body, a this expression evaluates to the instance that the method was\ncalled on. Or, more specifically, since methods are accessed and then invoked as\ntwo steps, it will refer to the object that the method was accessed from.

\n

That makes our job harder. Peep at:

\n
class Egotist {\n  speak() {\n    print this;\n  }\n}\n\nvar method = Egotist().speak;\nmethod();\n
\n

On the second-to-last line, we grab a reference to the speak() method off an\ninstance of the class. That returns a function, and that function needs to\nremember the instance it was pulled off of so that later, on the last line, it\ncan still find it when the function is called.

\n

We need to take this at the point that the method is accessed and attach it to\nthe function somehow so that it stays around as long as we need it to. Hmm . . . a\nway to store some extra data that hangs around a function, eh? That sounds an\nawful lot like a closure, doesn’t it?

\n

If we defined this as a sort of hidden variable in an environment that\nsurrounds the function returned when looking up a method, then uses of this in\nthe body would be able to find it later. LoxFunction already has the ability to\nhold on to a surrounding environment, so we have the machinery we need.

\n

Let’s walk through an example to see how it works:

\n
class Cake {\n  taste() {\n    var adjective = "delicious";\n    print "The " + this.flavor + " cake is " + adjective + "!";\n  }\n}\n\nvar cake = Cake();\ncake.flavor = "German chocolate";\ncake.taste(); // Prints "The German chocolate cake is delicious!".\n
\n

When we first evaluate the class definition, we create a LoxFunction for\ntaste(). Its closure is the environment surrounding the class, in this case\nthe global one. So the LoxFunction we store in the class’s method map looks\nlike so:

\"The\n

When we evaluate the cake.taste get expression, we create a new environment\nthat binds this to the object the method is accessed from (here, cake). Then\nwe make a new LoxFunction with the same code as the original one but using\nthat new environment as its closure.

\"The\n

This is the LoxFunction that gets returned when evaluating the get expression\nfor the method name. When that function is later called by a () expression,\nwe create an environment for the method body as usual.

\"Calling\n

The parent of the body environment is the environment we created earlier to bind\nthis to the current object. Thus any use of this inside the body\nsuccessfully resolves to that instance.

\n

Reusing our environment code for implementing this also takes care of\ninteresting cases where methods and functions interact, like:

\n
class Thing {\n  getCallback() {\n    fun localFunction() {\n      print this;\n    }\n\n    return localFunction;\n  }\n}\n\nvar callback = Thing().getCallback();\ncallback();\n
\n

In, say, JavaScript, it’s common to return a callback from inside a method. That\ncallback may want to hang on to and retain access to the original object—the\nthis value—that the method was associated with. Our existing support for\nclosures and environment chains should do all this correctly.

\n

Let’s code it up. The first step is adding new\nsyntax for this.

\n
      "Set      : Expr object, Token name, Expr value",\n
tool/GenerateAst.java
\nin main()
\n
      "This     : Token keyword",\n
      "Unary    : Token operator, Expr right",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

Parsing is simple since it’s a single token which our lexer already\nrecognizes as a reserved word.

\n
      return new Expr.Literal(previous().literal);\n    }\n
lox/Parser.java
\nin primary()
\n
\n\n    if (match(THIS)) return new Expr.This(previous());\n
\n\n    if (match(IDENTIFIER)) {\n
\n
lox/Parser.java, in primary()
\n\n

You can start to see how this works like a variable when we get to the\nresolver.

\n
lox/Resolver.java
\nadd after visitSetExpr()
\n
  @Override\n  public Void visitThisExpr(Expr.This expr) {\n    resolveLocal(expr, expr.keyword);\n    return null;\n  }\n\n
\n
lox/Resolver.java, add after visitSetExpr()
\n\n

We resolve it exactly like any other local variable using “this” as the name for\nthe “variable”. Of course, that’s not going to work right now, because “this”\nisn’t declared in any scope. Let’s fix that over in visitClassStmt().

\n
    define(stmt.name);\n\n
lox/Resolver.java
\nin visitClassStmt()
\n
    beginScope();\n    scopes.peek().put("this", true);\n\n
    for (Stmt.Function method : stmt.methods) {\n
\n
lox/Resolver.java, in visitClassStmt()
\n\n

Before we step in and start resolving the method bodies, we push a new scope and\ndefine “this” in it as if it were a variable. Then, when we’re done, we discard\nthat surrounding scope.

\n
    }\n\n
lox/Resolver.java
\nin visitClassStmt()
\n
    endScope();\n\n
    return null;\n
\n
lox/Resolver.java, in visitClassStmt()
\n\n

Now, whenever a this expression is encountered (at least inside a method) it\nwill resolve to a “local variable” defined in an implicit scope just outside of\nthe block for the method body.

\n

The resolver has a new scope for this, so the interpreter needs to create a\ncorresponding environment for it. Remember, we always have to keep the\nresolver’s scope chains and the interpreter’s linked environments in sync with\neach other. At runtime, we create the environment after we find the method on\nthe instance. We replace the previous line of code that simply returned the\nmethod’s LoxFunction with this:

\n
    LoxFunction method = klass.findMethod(name.lexeme);\n
lox/LoxInstance.java
\nin get()
\nreplace 1 line
\n
    if (method != null) return method.bind(this);\n
\n\n    throw new RuntimeError(name, \n        "Undefined property '" + name.lexeme + "'.");\n
\n
lox/LoxInstance.java, in get(), replace 1 line
\n\n

Note the new call to bind(). That looks like so:

\n
lox/LoxFunction.java
\nadd after LoxFunction()
\n
  LoxFunction bind(LoxInstance instance) {\n    Environment environment = new Environment(closure);\n    environment.define("this", instance);\n    return new LoxFunction(declaration, environment);\n  }\n
\n
lox/LoxFunction.java, add after LoxFunction()
\n\n

There isn’t much to it. We create a new environment nestled inside the method’s\noriginal closure. Sort of a closure-within-a-closure. When the method is called,\nthat will become the parent of the method body’s environment.

\n

We declare “this” as a variable in that environment and bind it to the given\ninstance, the instance that the method is being accessed from. Et voilà, the\nreturned LoxFunction now carries around its own little persistent world where\n“this” is bound to the object.

\n

The remaining task is interpreting those this expressions. Similar to the\nresolver, it is the same as interpreting a variable expression.

\n
lox/Interpreter.java
\nadd after visitSetExpr()
\n
  @Override\n  public Object visitThisExpr(Expr.This expr) {\n    return lookUpVariable(expr.keyword, expr);\n  }\n
\n
lox/Interpreter.java, add after visitSetExpr()
\n\n

Go ahead and give it a try using that cake example from earlier. With less than\ntwenty lines of code, our interpreter handles this inside methods even in all\nof the weird ways it can interact with nested classes, functions inside methods,\nhandles to methods, etc.

\n

12 . 6 . 1Invalid uses of this

\n

Wait a minute. What happens if you try to use this outside of a method? What\nabout:

\n
print this;\n
\n

Or:

\n
fun notAMethod() {\n  print this;\n}\n
\n

There is no instance for this to point to if you’re not in a method. We could\ngive it some default value like nil or make it a runtime error, but the user\nhas clearly made a mistake. The sooner they find and fix that mistake, the\nhappier they’ll be.

\n

Our resolution pass is a fine place to detect this error statically. It already\ndetects return statements outside of functions. We’ll do something similar for\nthis. In the vein of our existing FunctionType enum, we define a new ClassType\none.

\n
  }\n
lox/Resolver.java
\nadd after enum FunctionType
\n
\n\n  private enum ClassType {\n    NONE,\n    CLASS\n  }\n\n  private ClassType currentClass = ClassType.NONE;\n\n
  void resolve(List<Stmt> statements) {\n
\n
lox/Resolver.java, add after enum FunctionType
\n\n

Yes, it could be a Boolean. When we get to inheritance, it will get a third\nvalue, hence the enum right now. We also add a corresponding field,\ncurrentClass. Its value tells us if we are currently inside a class\ndeclaration while traversing the syntax tree. It starts out NONE which means\nwe aren’t in one.

\n

When we begin to resolve a class declaration, we change that.

\n
  public Void visitClassStmt(Stmt.Class stmt) {\n
lox/Resolver.java
\nin visitClassStmt()
\n
    ClassType enclosingClass = currentClass;\n    currentClass = ClassType.CLASS;\n\n
    declare(stmt.name);\n
\n
lox/Resolver.java, in visitClassStmt()
\n\n

As with currentFunction, we store the previous value of the field in a local\nvariable. This lets us piggyback onto the JVM to keep a stack of currentClass\nvalues. That way we don’t lose track of the previous value if one class nests\ninside another.

\n

Once the methods have been resolved, we “pop” that stack by restoring the old\nvalue.

\n
    endScope();\n\n
lox/Resolver.java
\nin visitClassStmt()
\n
    currentClass = enclosingClass;\n
    return null;\n
\n
lox/Resolver.java, in visitClassStmt()
\n\n

When we resolve a this expression, the currentClass field gives us the bit\nof data we need to report an error if the expression doesn’t occur nestled\ninside a method body.

\n
  public Void visitThisExpr(Expr.This expr) {\n
lox/Resolver.java
\nin visitThisExpr()
\n
    if (currentClass == ClassType.NONE) {\n      Lox.error(expr.keyword,\n          "Can't use 'this' outside of a class.");\n      return null;\n    }\n\n
    resolveLocal(expr, expr.keyword);\n
\n
lox/Resolver.java, in visitThisExpr()
\n\n

That should help users use this correctly, and it saves us from having to\nhandle misuse at runtime in the interpreter.

\n

12 . 7Constructors and Initializers

\n

We can do almost everything with classes now, and as we near the end of the\nchapter we find ourselves strangely focused on a beginning. Methods and fields\nlet us encapsulate state and behavior together so that an object always stays\nin a valid configuration. But how do we ensure a brand new object starts in a\ngood state?

\n

For that, we need constructors. I find them one of the trickiest parts of a\nlanguage to design, and if you peer closely at most other languages, you’ll see\ncracks around object construction where the seams of\nthe design don’t quite fit together perfectly. Maybe there’s something\nintrinsically messy about the moment of birth.

\n\n

“Constructing” an object is actually a pair of operations:

\n
    \n
  1. \n

    The runtime allocates the memory required for\na fresh instance. In most languages, this operation is at a fundamental\nlevel beneath what user code is able to access.

    \n
    2. \n
  2. \n

    Then, a user-provided chunk of code is called which initializes the\nunformed object.

    \n
    3. \n
\n

The latter is what we tend to think of when we hear “constructor”, but the\nlanguage itself has usually done some groundwork for us before we get to that\npoint. In fact, our Lox interpreter already has that covered when it creates a\nnew LoxInstance object.

\n

We’ll do the remaining part—user-defined initialization—now. Languages\nhave a variety of notations for the chunk of code that sets up a new object for\na class. C++, Java, and C# use a method whose name matches the class name. Ruby\nand Python call it init(). The latter is nice and short, so we’ll do that.

\n

In LoxClass’s implementation of LoxCallable, we add a few more lines.

\n
                     List<Object> arguments) {\n    LoxInstance instance = new LoxInstance(this);\n
lox/LoxClass.java
\nin call()
\n
    LoxFunction initializer = findMethod("init");\n    if (initializer != null) {\n      initializer.bind(instance).call(interpreter, arguments);\n    }\n\n
    return instance;\n
\n
lox/LoxClass.java, in call()
\n\n

When a class is called, after the LoxInstance is created, we look for an “init”\nmethod. If we find one, we immediately bind and invoke it just like a normal\nmethod call. The argument list is forwarded along.

\n

That argument list means we also need to tweak how a class declares its arity.

\n
  public int arity() {\n
lox/LoxClass.java
\nin arity()
\nreplace 1 line
\n
    LoxFunction initializer = findMethod("init");\n    if (initializer == null) return 0;\n    return initializer.arity();\n
  }\n
\n
lox/LoxClass.java, in arity(), replace 1 line
\n\n

If there is an initializer, that method’s arity determines how many arguments\nyou must pass when you call the class itself. We don’t require a class to\ndefine an initializer, though, as a convenience. If you don’t have an\ninitializer, the arity is still zero.

\n

That’s basically it. Since we bind the init() method before we call it, it has\naccess to this inside its body. That, along with the arguments passed to the\nclass, are all you need to be able to set up the new instance however you\ndesire.

\n

12 . 7 . 1Invoking init() directly

\n

As usual, exploring this new semantic territory rustles up a few weird\ncreatures. Consider:

\n
class Foo {\n  init() {\n    print this;\n  }\n}\n\nvar foo = Foo();\nprint foo.init();\n
\n

Can you “re-initialize” an object by directly calling its init() method? If\nyou do, what does it return? A reasonable answer\nwould be nil since that’s what it appears the body returns.

\n

However—and I generally dislike compromising to satisfy the\nimplementation—it will make clox’s implementation of constructors much\neasier if we say that init() methods always return this, even when\ndirectly called. In order to keep jlox compatible with that, we add a little\nspecial case code in LoxFunction.

\n\n
      return returnValue.value;\n    }\n
lox/LoxFunction.java
\nin call()
\n
\n\n    if (isInitializer) return closure.getAt(0, "this");\n
    return null;\n
\n
lox/LoxFunction.java, in call()
\n\n

If the function is an initializer, we override the actual return value and\nforcibly return this. That relies on a new isInitializer field.

\n
  private final Environment closure;\n\n
lox/LoxFunction.java
\nin class LoxFunction
\nreplace 1 line
\n
  private final boolean isInitializer;\n\n  LoxFunction(Stmt.Function declaration, Environment closure,\n              boolean isInitializer) {\n    this.isInitializer = isInitializer;\n
    this.closure = closure;\n    this.declaration = declaration;\n
\n
lox/LoxFunction.java, in class LoxFunction, replace 1 line
\n\n

We can’t simply see if the name of the LoxFunction is “init” because the user\ncould have defined a function with that name. In that case, there is no\nthis to return. To avoid that weird edge case, we’ll directly store whether\nthe LoxFunction represents an initializer method. That means we need to go back\nand fix the few places where we create LoxFunctions.

\n
  public Void visitFunctionStmt(Stmt.Function stmt) {\n
lox/Interpreter.java
\nin visitFunctionStmt()
\nreplace 1 line
\n
    LoxFunction function = new LoxFunction(stmt, environment,\n                                           false);\n
    environment.define(stmt.name.lexeme, function);\n
\n
lox/Interpreter.java, in visitFunctionStmt(), replace 1 line
\n\n

For actual function declarations, isInitializer is always false. For methods,\nwe check the name.

\n
    for (Stmt.Function method : stmt.methods) {\n
lox/Interpreter.java
\nin visitClassStmt()
\nreplace 1 line
\n
      LoxFunction function = new LoxFunction(method, environment,\n          method.name.lexeme.equals("init"));\n
      methods.put(method.name.lexeme, function);\n
\n
lox/Interpreter.java, in visitClassStmt(), replace 1 line
\n\n

And then in bind() where we create the closure that binds this to a method,\nwe pass along the original method’s value.

\n
    environment.define("this", instance);\n
lox/LoxFunction.java
\nin bind()
\nreplace 1 line
\n
    return new LoxFunction(declaration, environment,\n                           isInitializer);\n
  }\n
\n
lox/LoxFunction.java, in bind(), replace 1 line
\n\n

12 . 7 . 2Returning from init()

\n

We aren’t out of the woods yet. We’ve been assuming that a user-written\ninitializer doesn’t explicitly return a value because most constructors don’t.\nWhat should happen if a user tries:

\n
class Foo {\n  init() {\n    return "something else";\n  }\n}\n
\n

It’s definitely not going to do what they want, so we may as well make it a\nstatic error. Back in the resolver, we add another case to FunctionType.

\n
    FUNCTION,\n
lox/Resolver.java
\nin enum FunctionType
\n
    INITIALIZER,\n
    METHOD\n
\n
lox/Resolver.java, in enum FunctionType
\n\n

We use the visited method’s name to determine if we’re resolving an initializer\nor not.

\n
      FunctionType declaration = FunctionType.METHOD;\n
lox/Resolver.java
\nin visitClassStmt()
\n
      if (method.name.lexeme.equals("init")) {\n        declaration = FunctionType.INITIALIZER;\n      }\n\n
      resolveFunction(method, declaration); \n
\n
lox/Resolver.java, in visitClassStmt()
\n\n

When we later traverse into a return statement, we check that field and make\nit an error to return a value from inside an init() method.

\n
    if (stmt.value != null) {\n
lox/Resolver.java
\nin visitReturnStmt()
\n
      if (currentFunction == FunctionType.INITIALIZER) {\n        Lox.error(stmt.keyword,\n            "Can't return a value from an initializer.");\n      }\n\n
      resolve(stmt.value);\n
\n
lox/Resolver.java, in visitReturnStmt()
\n\n

We’re still not done. We statically disallow returning a value from an\ninitializer, but you can still use an empty early return.

\n
class Foo {\n  init() {\n    return;\n  }\n}\n
\n

That is actually kind of useful sometimes, so we don’t want to disallow it\nentirely. Instead, it should return this instead of nil. That’s an easy fix\nover in LoxFunction.

\n
    } catch (Return returnValue) {\n
lox/LoxFunction.java
\nin call()
\n
      if (isInitializer) return closure.getAt(0, "this");\n\n
      return returnValue.value;\n
\n
lox/LoxFunction.java, in call()
\n\n

If we’re in an initializer and execute a return statement, instead of\nreturning the value (which will always be nil), we again return this.

\n

Phew! That was a whole list of tasks but our reward is that our little\ninterpreter has grown an entire programming paradigm. Classes, methods, fields,\nthis, and constructors. Our baby language is looking awfully grown-up.

\n
\n

Challenges

\n
    \n
  1. \n

    We have methods on instances, but there is no way to define “static” methods\nthat can be called directly on the class object itself. Add support for\nthem. Use a class keyword preceding the method to indicate a static method\nthat hangs off the class object.

    \n
    class Math {\n  class square(n) {\n    return n * n;\n  }\n}\n\nprint Math.square(3); // Prints "9".\n
    \n

    You can solve this however you like, but the “metaclasses” used by\nSmalltalk and Ruby are a particularly elegant approach. Hint: Make LoxClass\nextend LoxInstance and go from there.

    \n
    2. \n
  2. \n

    Most modern languages support “getters” and “setters”—members on a class\nthat look like field reads and writes but that actually execute user-defined\ncode. Extend Lox to support getter methods. These are declared without a\nparameter list. The body of the getter is executed when a property with that\nname is accessed.

    \n
    class Circle {\n  init(radius) {\n    this.radius = radius;\n  }\n\n  area {\n    return 3.141592653 * this.radius * this.radius;\n  }\n}\n\nvar circle = Circle(4);\nprint circle.area; // Prints roughly "50.2655".\n
    \n
    3. \n
  3. \n

    Python and JavaScript allow you to freely access an object’s fields from\noutside of its own methods. Ruby and Smalltalk encapsulate instance state.\nOnly methods on the class can access the raw fields, and it is up to the\nclass to decide which state is exposed. Most statically typed languages\noffer modifiers like private and public to control which parts of a\nclass are externally accessible on a per-member basis.

    \n

    What are the trade-offs between these approaches and why might a language\nprefer one or the other?

    \n
    4. \n
\n
\n
\n

Design Note: Prototypes and Power

\n

In this chapter, we introduced two new runtime entities, LoxClass and\nLoxInstance. The former is where behavior for objects lives, and the latter is\nfor state. What if you could define methods right on a single object, inside\nLoxInstance? In that case, we wouldn’t need LoxClass at all. LoxInstance would\nbe a complete package for defining the behavior and state of an object.

\n

We’d still want some way, without classes, to reuse behavior across multiple\ninstances. We could let a LoxInstance delegate directly to another\nLoxInstance to reuse its fields and methods, sort of like inheritance.

\n

Users would model their program as a constellation of objects, some of which\ndelegate to each other to reflect commonality. Objects used as delegates\nrepresent “canonical” or “prototypical” objects that others refine. The result\nis a simpler runtime with only a single internal construct, LoxInstance.

\n

That’s where the name prototypes comes from for this paradigm. It\nwas invented by David Ungar and Randall Smith in a language called Self.\nThey came up with it by starting with Smalltalk and following the above mental\nexercise to see how much they could pare it down.

\n

Prototypes were an academic curiosity for a long time, a fascinating one that\ngenerated interesting research but didn’t make a dent in the larger world of\nprogramming. That is, until Brendan Eich crammed prototypes into JavaScript,\nwhich then promptly took over the world. Many (many) words have been written about prototypes in JavaScript.\nWhether that shows that prototypes are brilliant or confusing—or both!—is\nan open question.

\n\n

I won’t get into whether or not I think prototypes are a good idea for a\nlanguage. I’ve made languages that are prototypal and\nclass-based, and my opinions of both are complex. What I want to discuss\nis the role of simplicity in a language.

\n

Prototypes are simpler than classes—less code for the language implementer to\nwrite, and fewer concepts for the user to learn and understand. Does that make\nthem better? We language nerds have a tendency to fetishize minimalism.\nPersonally, I think simplicity is only part of the equation. What we really want\nto give the user is power, which I define as:

\n
power = breadth × ease ÷ complexity\n
\n

None of these are precise numeric measures. I’m using math as analogy here, not\nactual quantification.

\n
    \n
  • \n

    Breadth is the range of different things the language lets you express.\nC has a lot of breadth—it’s been used for everything from operating\nsystems to user applications to games. Domain-specific languages like\nAppleScript and Matlab have less breadth.

    \n
    • \n
  • \n

    Ease is how little effort it takes to make the language do what you\nwant. “Usability” might be another term, though it carries more baggage than\nI want to bring in. “Higher-level” languages tend to have more ease than\n“lower-level” ones. Most languages have a “grain” to them where some things\nfeel easier to express than others.

    \n
    • \n
  • \n

    Complexity is how big the language (including its runtime, core libraries,\ntools, ecosystem, etc.) is. People talk about how many pages are in a\nlanguage’s spec, or how many keywords it has. It’s how much the user has to\nload into their wetware before they can be productive in the system. It is\nthe antonym of simplicity.

    \n
    • \n
\n

Reducing complexity does increase power. The smaller the denominator, the\nlarger the resulting value, so our intuition that simplicity is good is valid.\nHowever, when reducing complexity, we must take care not to sacrifice breadth or\nease in the process, or the total power may go down. Java would be a strictly\nsimpler language if it removed strings, but it probably wouldn’t handle text\nmanipulation tasks well, nor would it be as easy to get things done.

\n

The art, then, is finding accidental complexity that can be omitted—language features and interactions that don’t carry their weight by increasing\nthe breadth or ease of using the language.

\n

If users want to express their program in terms of categories of objects, then\nbaking classes into the language increases the ease of doing that, hopefully by\na large enough margin to pay for the added complexity. But if that isn’t how\nusers are using your language, then by all means leave classes out.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/closures.html", "content": "\n\n\n\nClosures · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
25
\n

Closures

\n\n
\n

As the man said, for every complex problem there’s a simple solution, and it’s\nwrong.

\n

Umberto Eco, Foucault’s Pendulum

\n
\n

Thanks to our diligent labor in the last chapter, we have a virtual\nmachine with working functions. What it lacks is closures. Aside from global\nvariables, which are their own breed of animal, a function has no way to\nreference a variable declared outside of its own body.

\n
var x = "global";\nfun outer() {\n  var x = "outer";\n  fun inner() {\n    print x;\n  }\n  inner();\n}\nouter();\n
\n

Run this example now and it prints “global”. It’s supposed to print “outer”. To\nfix this, we need to include the entire lexical scope of all surrounding\nfunctions when resolving a variable.

\n

This problem is harder in clox than it was in jlox because our bytecode VM\nstores locals on a stack. We used a stack because I claimed locals have stack\nsemantics—variables are discarded in the reverse order that they are created.\nBut with closures, that’s only mostly true.

\n
fun makeClosure() {\n  var local = "local";\n  fun closure() {\n    print local;\n  }\n  return closure;\n}\n\nvar closure = makeClosure();\nclosure();\n
\n

The outer function makeClosure() declares a variable, local. It also creates\nan inner function, closure() that captures that variable. Then makeClosure()\nreturns a reference to that function. Since the closure escapes while holding on to the local variable, local must\noutlive the function call where it was created.

\n\n

We could solve this problem by dynamically allocating memory for all local\nvariables. That’s what jlox does by putting everything in those Environment\nobjects that float around in Java’s heap. But we don’t want to. Using a stack is really fast. Most local variables are not\ncaptured by closures and do have stack semantics. It would suck to make all of\nthose slower for the benefit of the rare local that is captured.

\n\n

This means a more complex approach than we used in our Java interpreter. Because\nsome locals have very different lifetimes, we will have two implementation\nstrategies. For locals that aren’t used in closures, we’ll keep them just as\nthey are on the stack. When a local is captured by a closure, we’ll adopt\nanother solution that lifts them onto the heap where they can live as long as\nneeded.

\n

Closures have been around since the early Lisp days when bytes of memory and CPU\ncycles were more precious than emeralds. Over the intervening decades, hackers\ndevised all manner of ways to compile closures to\noptimized runtime representations. Some are more efficient but require a more\ncomplex compilation process than we could easily retrofit into clox.

\n\n

The technique I explain here comes from the design of the Lua VM. It is fast,\nparsimonious with memory, and implemented with relatively little code. Even more\nimpressive, it fits naturally into the single-pass compilers clox and Lua both\nuse. It is somewhat intricate, though. It might take a while before all the\npieces click together in your mind. We’ll build them one step at a time, and\nI’ll try to introduce the concepts in stages.

\n

25 . 1Closure Objects

\n

Our VM represents functions at runtime using ObjFunction. These objects are\ncreated by the front end during compilation. At runtime, all the VM does is load\nthe function object from a constant table and bind it to a name. There is no\noperation to “create” a function at runtime. Much like string and number literals, they are constants instantiated purely at\ncompile time.

\n\n

That made sense because all of the data that composes a function is known at\ncompile time: the chunk of bytecode compiled from the function’s body, and the\nconstants used in the body. Once we introduce closures, though, that\nrepresentation is no longer sufficient. Take a gander at:

\n
fun makeClosure(value) {\n  fun closure() {\n    print value;\n  }\n  return closure;\n}\n\nvar doughnut = makeClosure("doughnut");\nvar bagel = makeClosure("bagel");\ndoughnut();\nbagel();\n
\n

The makeClosure() function defines and returns a function. We call it twice\nand get two closures back. They are created by the same nested function\ndeclaration, closure, but close over different values. When we call the two\nclosures, each prints a different string. That implies we need some runtime\nrepresentation for a closure that captures the local variables surrounding the\nfunction as they exist when the function declaration is executed, not just\nwhen it is compiled.

\n

We’ll work our way up to capturing variables, but a good first step is defining\nthat object representation. Our existing ObjFunction type represents the “raw” compile-time state of a function declaration, since all\nclosures created from a single declaration share the same code and constants. At\nruntime, when we execute a function declaration, we wrap the ObjFunction in a\nnew ObjClosure structure. The latter has a reference to the underlying bare\nfunction along with runtime state for the variables the function closes over.

\n\"An\n

We’ll wrap every function in an ObjClosure, even if the function doesn’t\nactually close over and capture any surrounding local variables. This is a\nlittle wasteful, but it simplifies the VM because we can always assume that the\nfunction we’re calling is an ObjClosure. That new struct starts out like this:

\n
object.h
\nadd after struct ObjString
\n
typedef struct {\n  Obj obj;\n  ObjFunction* function;\n} ObjClosure;\n
\n
object.h, add after struct ObjString
\n\n

Right now, it simply points to an ObjFunction and adds the necessary object\nheader stuff. Grinding through the usual ceremony for adding a new object type\nto clox, we declare a C function to create a new closure.

\n
} ObjClosure;\n\n
object.h
\nadd after struct ObjClosure
\n
ObjClosure* newClosure(ObjFunction* function);\n
ObjFunction* newFunction();\n
\n
object.h, add after struct ObjClosure
\n\n

Then we implement it here:

\n
object.c
\nadd after allocateObject()
\n
ObjClosure* newClosure(ObjFunction* function) {\n  ObjClosure* closure = ALLOCATE_OBJ(ObjClosure, OBJ_CLOSURE);\n  closure->function = function;\n  return closure;\n}\n
\n
object.c, add after allocateObject()
\n\n

It takes a pointer to the ObjFunction it wraps. It also initializes the type\nfield to a new type.

\n
typedef enum {\n
object.h
\nin enum ObjType
\n
  OBJ_CLOSURE,\n
  OBJ_FUNCTION,\n
\n
object.h, in enum ObjType
\n\n

And when we’re done with a closure, we release its memory.

\n
  switch (object->type) {\n
memory.c
\nin freeObject()
\n
    case OBJ_CLOSURE: {\n      FREE(ObjClosure, object);\n      break;\n    }\n
    case OBJ_FUNCTION: {\n
\n
memory.c, in freeObject()
\n\n

We free only the ObjClosure itself, not the ObjFunction. That’s because the\nclosure doesn’t own the function. There may be multiple closures that all\nreference the same function, and none of them claims any special privilege over\nit. We can’t free the ObjFunction until all objects referencing it are gone—including even the surrounding function whose constant table contains it.\nTracking that sounds tricky, and it is! That’s why we’ll write a garbage\ncollector soon to manage it for us.

\n

We also have the usual macros for checking a value’s\ntype.

\n\n
#define OBJ_TYPE(value)        (AS_OBJ(value)->type)\n\n
object.h
\n
#define IS_CLOSURE(value)      isObjType(value, OBJ_CLOSURE)\n
#define IS_FUNCTION(value)     isObjType(value, OBJ_FUNCTION)\n
\n
object.h
\n\n

And to cast a value:

\n
#define IS_STRING(value)       isObjType(value, OBJ_STRING)\n\n
object.h
\n
#define AS_CLOSURE(value)      ((ObjClosure*)AS_OBJ(value))\n
#define AS_FUNCTION(value)     ((ObjFunction*)AS_OBJ(value))\n
\n
object.h
\n\n

Closures are first-class objects, so you can print them.

\n
  switch (OBJ_TYPE(value)) {\n
object.c
\nin printObject()
\n
    case OBJ_CLOSURE:\n      printFunction(AS_CLOSURE(value)->function);\n      break;\n
    case OBJ_FUNCTION:\n
\n
object.c, in printObject()
\n\n

They display exactly as ObjFunction does. From the user’s perspective, the\ndifference between ObjFunction and ObjClosure is purely a hidden implementation\ndetail. With that out of the way, we have a working but empty representation for\nclosures.

\n

25 . 1 . 1Compiling to closure objects

\n

We have closure objects, but our VM never creates them. The next step is getting\nthe compiler to emit instructions to tell the runtime when to create a new\nObjClosure to wrap a given ObjFunction. This happens right at the end of a\nfunction declaration.

\n
  ObjFunction* function = endCompiler();\n
compiler.c
\nin function()
\nreplace 1 line
\n
  emitBytes(OP_CLOSURE, makeConstant(OBJ_VAL(function)));\n
}\n
\n
compiler.c, in function(), replace 1 line
\n\n

Before, the final bytecode for a function declaration was a single OP_CONSTANT\ninstruction to load the compiled function from the surrounding function’s\nconstant table and push it onto the stack. Now we have a new instruction.

\n
  OP_CALL,\n
chunk.h
\nin enum OpCode
\n
  OP_CLOSURE,\n
  OP_RETURN,\n
\n
chunk.h, in enum OpCode
\n\n

Like OP_CONSTANT, it takes a single operand that represents a constant table\nindex for the function. But when we get over to the runtime implementation, we\ndo something more interesting.

\n

First, let’s be diligent VM hackers and slot in disassembler support for the\ninstruction.

\n
    case OP_CALL:\n      return byteInstruction("OP_CALL", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_CLOSURE: {\n      offset++;\n      uint8_t constant = chunk->code[offset++];\n      printf("%-16s %4d ", "OP_CLOSURE", constant);\n      printValue(chunk->constants.values[constant]);\n      printf("\\n");\n      return offset;\n    }\n
    case OP_RETURN:\n
\n
debug.c, in disassembleInstruction()
\n\n

There’s more going on here than we usually have in the disassembler. By the end\nof the chapter, you’ll discover that OP_CLOSURE is quite an unusual\ninstruction. It’s straightforward right now—just a single byte operand—but\nwe’ll be adding to it. This code here anticipates that future.

\n

25 . 1 . 2Interpreting function declarations

\n

Most of the work we need to do is in the runtime. We have to handle the new\ninstruction, naturally. But we also need to touch every piece of code in the VM\nthat works with ObjFunction and change it to use ObjClosure instead—function\ncalls, call frames, etc. We’ll start with the instruction, though.

\n
      }\n
vm.c
\nin run()
\n
      case OP_CLOSURE: {\n        ObjFunction* function = AS_FUNCTION(READ_CONSTANT());\n        ObjClosure* closure = newClosure(function);\n        push(OBJ_VAL(closure));\n        break;\n      }\n
      case OP_RETURN: {\n
\n
vm.c, in run()
\n\n

Like the OP_CONSTANT instruction we used before, first we load the compiled\nfunction from the constant table. The difference now is that we wrap that\nfunction in a new ObjClosure and push the result onto the stack.

\n

Once you have a closure, you’ll eventually want to call it.

\n
    switch (OBJ_TYPE(callee)) {\n
vm.c
\nin callValue()
\nreplace 2 lines
\n
      case OBJ_CLOSURE:\n        return call(AS_CLOSURE(callee), argCount);\n
      case OBJ_NATIVE: {\n
\n
vm.c, in callValue(), replace 2 lines
\n\n

We remove the code for calling objects whose type is OBJ_FUNCTION. Since we\nwrap all functions in ObjClosures, the runtime will never try to invoke a bare\nObjFunction anymore. Those objects live only in constant tables and get\nimmediately wrapped in closures before anything else\nsees them.

\n\n

We replace the old code with very similar code for calling a closure instead.\nThe only difference is the type of object we pass to call(). The real changes\nare over in that function. First, we update its signature.

\n
vm.c
\nfunction call()
\nreplace 1 line
\n
static bool call(ObjClosure* closure, int argCount) {\n
  if (argCount != function->arity) {\n
\n
vm.c, function call(), replace 1 line
\n\n

Then, in the body, we need to fix everything that referenced the function to\nhandle the fact that we’ve introduced a layer of indirection. We start with the\narity checking:

\n
static bool call(ObjClosure* closure, int argCount) {\n
vm.c
\nin call()
\nreplace 3 lines
\n
  if (argCount != closure->function->arity) {\n    runtimeError("Expected %d arguments but got %d.",\n        closure->function->arity, argCount);\n
    return false;\n
\n
vm.c, in call(), replace 3 lines
\n\n

The only change is that we unwrap the closure to get to the underlying function.\nThe next thing call() does is create a new CallFrame. We change that code to\nstore the closure in the CallFrame and get the bytecode pointer from the\nclosure’s function.

\n
  CallFrame* frame = &vm.frames[vm.frameCount++];\n
vm.c
\nin call()
\nreplace 2 lines
\n
  frame->closure = closure;\n  frame->ip = closure->function->chunk.code;\n
  frame->slots = vm.stackTop - argCount - 1;\n
\n
vm.c, in call(), replace 2 lines
\n\n

This necessitates changing the declaration of CallFrame too.

\n
typedef struct {\n
vm.h
\nin struct CallFrame
\nreplace 1 line
\n
  ObjClosure* closure;\n
  uint8_t* ip;\n
\n
vm.h, in struct CallFrame, replace 1 line
\n\n

That change triggers a few other cascading changes. Every place in the VM that\naccessed CallFrame’s function needs to use a closure instead. First, the macro\nfor reading a constant from the current function’s constant table:

\n
    (uint16_t)((frame->ip[-2] << 8) | frame->ip[-1]))\n\n
vm.c
\nin run()
\nreplace 2 lines
\n
#define READ_CONSTANT() \\\n    (frame->closure->function->chunk.constants.values[READ_BYTE()])\n
\n\n#define READ_STRING() AS_STRING(READ_CONSTANT())\n
\n
vm.c, in run(), replace 2 lines
\n\n

When DEBUG_TRACE_EXECUTION is enabled, it needs to get to the chunk from the\nclosure.

\n
    printf("\\n");\n
vm.c
\nin run()
\nreplace 2 lines
\n
    disassembleInstruction(&frame->closure->function->chunk,\n        (int)(frame->ip - frame->closure->function->chunk.code));\n
#endif\n
\n
vm.c, in run(), replace 2 lines
\n\n

Likewise when reporting a runtime error:

\n
    CallFrame* frame = &vm.frames[i];\n
vm.c
\nin runtimeError()
\nreplace 1 line
\n
    ObjFunction* function = frame->closure->function;\n
    size_t instruction = frame->ip - function->chunk.code - 1;\n
\n
vm.c, in runtimeError(), replace 1 line
\n\n

Almost there. The last piece is the blob of code that sets up the very first\nCallFrame to begin executing the top-level code for a Lox script.

\n
  push(OBJ_VAL(function));\n
vm.c
\nin interpret()
\nreplace 1 line
\n
  ObjClosure* closure = newClosure(function);\n  pop();\n  push(OBJ_VAL(closure));\n  call(closure, 0);\n
\n\n  return run();\n
\n
vm.c, in interpret(), replace 1 line
\n\n

The compiler still returns a raw ObjFunction when\ncompiling a script. That’s fine, but it means we need to wrap it in an\nObjClosure here, before the VM can execute it.

\n\n

We are back to a working interpreter. The user can’t tell any difference, but\nthe compiler now generates code telling the VM to create a closure for each\nfunction declaration. Every time the VM executes a function declaration, it\nwraps the ObjFunction in a new ObjClosure. The rest of the VM now handles those\nObjClosures floating around. That’s the boring stuff out of the way. Now we’re\nready to make these closures actually do something.

\n

25 . 2Upvalues

\n

Our existing instructions for reading and writing local variables are limited to\na single function’s stack window. Locals from a surrounding function are outside\nof the inner function’s window. We’re going to need some new instructions.

\n

The easiest approach might be an instruction that takes a relative stack slot\noffset that can reach before the current function’s window. That would work if\nclosed-over variables were always on the stack. But as we saw earlier, these\nvariables sometimes outlive the function where they are declared. That means\nthey won’t always be on the stack.

\n

The next easiest approach, then, would be to take any local variable that gets\nclosed over and have it always live on the heap. When the local variable\ndeclaration in the surrounding function is executed, the VM would allocate\nmemory for it dynamically. That way it could live as long as needed.

\n

This would be a fine approach if clox didn’t have a single-pass compiler. But\nthat restriction we chose in our implementation makes things harder. Take a look\nat this example:

\n
fun outer() {\n  var x = 1;    // (1)\n  x = 2;        // (2)\n  fun inner() { // (3)\n    print x;\n  }\n  inner();\n}\n
\n

Here, the compiler compiles the declaration of x at (1) and emits code for\nthe assignment at (2). It does that before reaching the declaration of\ninner() at (3) and discovering that x is in fact closed over. We don’t\nhave an easy way to go back and fix that already-emitted code to treat x\nspecially. Instead, we want a solution that allows a closed-over variable to\nlive on the stack exactly like a normal local variable until the point that it\nis closed over.

\n

Fortunately, thanks to the Lua dev team, we have a solution. We use a level of\nindirection that they call an upvalue. An upvalue refers to a local variable\nin an enclosing function. Every closure maintains an array of upvalues, one for\neach surrounding local variable that the closure uses.

\n

The upvalue points back into the stack to where the variable it captured lives.\nWhen the closure needs to access a closed-over variable, it goes through the\ncorresponding upvalue to reach it. When a function declaration is first executed\nand we create a closure for it, the VM creates the array of upvalues and wires\nthem up to “capture” the surrounding local variables that the closure needs.

\n

For example, if we throw this program at clox,

\n
{\n  var a = 3;\n  fun f() {\n    print a;\n  }\n}\n
\n

the compiler and runtime will conspire together to build up a set of objects in\nmemory like this:

\"The\n

That might look overwhelming, but fear not. We’ll work our way through it. The\nimportant part is that upvalues serve as the layer of indirection needed to\ncontinue to find a captured local variable even after it moves off the stack.\nBut before we get to all that, let’s focus on compiling captured variables.

\n

25 . 2 . 1Compiling upvalues

\n

As usual, we want to do as much work as possible during compilation to keep\nexecution simple and fast. Since local variables are lexically scoped in Lox, we\nhave enough knowledge at compile time to resolve which surrounding local\nvariables a function accesses and where those locals are declared. That, in\nturn, means we know how many upvalues a closure needs, which variables they\ncapture, and which stack slots contain those variables in the declaring\nfunction’s stack window.

\n

Currently, when the compiler resolves an identifier, it walks the block scopes\nfor the current function from innermost to outermost. If we don’t find the\nvariable in that function, we assume the variable must be a global. We don’t\nconsider the local scopes of enclosing functions—they get skipped right over.\nThe first change, then, is inserting a resolution step for those outer local\nscopes.

\n
  if (arg != -1) {\n    getOp = OP_GET_LOCAL;\n    setOp = OP_SET_LOCAL;\n
compiler.c
\nin namedVariable()
\n
  } else if ((arg = resolveUpvalue(current, &name)) != -1) {\n    getOp = OP_GET_UPVALUE;\n    setOp = OP_SET_UPVALUE;\n
  } else {\n
\n
compiler.c, in namedVariable()
\n\n

This new resolveUpvalue() function looks for a local variable declared in any\nof the surrounding functions. If it finds one, it returns an “upvalue index” for\nthat variable. (We’ll get into what that means later.) Otherwise, it returns -1\nto indicate the variable wasn’t found. If it was found, we use these two new\ninstructions for reading or writing to the variable through its upvalue:

\n
  OP_SET_GLOBAL,\n
chunk.h
\nin enum OpCode
\n
  OP_GET_UPVALUE,\n  OP_SET_UPVALUE,\n
  OP_EQUAL,\n
\n
chunk.h, in enum OpCode
\n\n

We’re implementing this sort of top-down, so I’ll show you how these work at\nruntime soon. The part to focus on now is how the compiler actually resolves the\nidentifier.

\n
compiler.c
\nadd after resolveLocal()
\n
static int resolveUpvalue(Compiler* compiler, Token* name) {\n  if (compiler->enclosing == NULL) return -1;\n\n  int local = resolveLocal(compiler->enclosing, name);\n  if (local != -1) {\n    return addUpvalue(compiler, (uint8_t)local, true);\n  }\n\n  return -1;\n}\n
\n
compiler.c, add after resolveLocal()
\n\n

We call this after failing to resolve a local variable in the current function’s\nscope, so we know the variable isn’t in the current compiler. Recall that\nCompiler stores a pointer to the Compiler for the enclosing function, and these\npointers form a linked chain that goes all the way to the root Compiler for the\ntop-level code. Thus, if the enclosing Compiler is NULL, we know we’ve reached\nthe outermost function without finding a local variable. The variable must be\nglobal, so we return -1.

\n\n

Otherwise, we try to resolve the identifier as a local variable in the\nenclosing compiler. In other words, we look for it right outside the current\nfunction. For example:

\n
fun outer() {\n  var x = 1;\n  fun inner() {\n    print x; // (1)\n  }\n  inner();\n}\n
\n

When compiling the identifier expression at (1), resolveUpvalue() looks for\na local variable x declared in outer(). If found—like it is in this\nexample—then we’ve successfully resolved the variable. We create an upvalue\nso that the inner function can access the variable through that. The upvalue is\ncreated here:

\n
compiler.c
\nadd after resolveLocal()
\n
static int addUpvalue(Compiler* compiler, uint8_t index,\n                      bool isLocal) {\n  int upvalueCount = compiler->function->upvalueCount;\n  compiler->upvalues[upvalueCount].isLocal = isLocal;\n  compiler->upvalues[upvalueCount].index = index;\n  return compiler->function->upvalueCount++;\n}\n
\n
compiler.c, add after resolveLocal()
\n\n

The compiler keeps an array of upvalue structures to track the closed-over\nidentifiers that it has resolved in the body of each function. Remember how the\ncompiler’s Local array mirrors the stack slot indexes where locals live at\nruntime? This new upvalue array works the same way. The indexes in the\ncompiler’s array match the indexes where upvalues will live in the ObjClosure at\nruntime.

\n

This function adds a new upvalue to that array. It also keeps track of the\nnumber of upvalues the function uses. It stores that count directly in the\nObjFunction itself because we’ll also need that\nnumber for use at runtime.

\n\n

The index field tracks the closed-over local variable’s slot index. That way\nthe compiler knows which variable in the enclosing function needs to be\ncaptured. We’ll circle back to what that isLocal field is for before too long.\nFinally, addUpvalue() returns the index of the created upvalue in the\nfunction’s upvalue list. That index becomes the operand to the OP_GET_UPVALUE\nand OP_SET_UPVALUE instructions.

\n

That’s the basic idea for resolving upvalues, but the function isn’t fully\nbaked. A closure may reference the same variable in a surrounding function\nmultiple times. In that case, we don’t want to waste time and memory creating a\nseparate upvalue for each identifier expression. To fix that, before we add a\nnew upvalue, we first check to see if the function already has an upvalue that\ncloses over that variable.

\n
  int upvalueCount = compiler->function->upvalueCount;\n
compiler.c
\nin addUpvalue()
\n
\n\n  for (int i = 0; i < upvalueCount; i++) {\n    Upvalue* upvalue = &compiler->upvalues[i];\n    if (upvalue->index == index && upvalue->isLocal == isLocal) {\n      return i;\n    }\n  }\n\n
  compiler->upvalues[upvalueCount].isLocal = isLocal;\n
\n
compiler.c, in addUpvalue()
\n\n

If we find an upvalue in the array whose slot index matches the one we’re\nadding, we just return that upvalue index and reuse it. Otherwise, we fall\nthrough and add the new upvalue.

\n

These two functions access and modify a bunch of new state, so let’s define\nthat. First, we add the upvalue count to ObjFunction.

\n
  int arity;\n
object.h
\nin struct ObjFunction
\n
  int upvalueCount;\n
  Chunk chunk;\n
\n
object.h, in struct ObjFunction
\n\n

We’re conscientious C programmers, so we zero-initialize that when an\nObjFunction is first allocated.

\n
  function->arity = 0;\n
object.c
\nin newFunction()
\n
  function->upvalueCount = 0;\n
  function->name = NULL;\n
\n
object.c, in newFunction()
\n\n

In the compiler, we add a field for the upvalue array.

\n
  int localCount;\n
compiler.c
\nin struct Compiler
\n
  Upvalue upvalues[UINT8_COUNT];\n
  int scopeDepth;\n
\n
compiler.c, in struct Compiler
\n\n

For simplicity, I gave it a fixed size. The OP_GET_UPVALUE and\nOP_SET_UPVALUE instructions encode an upvalue index using a single byte\noperand, so there’s a restriction on how many upvalues a function can have—how many unique variables it can close over. Given that, we can afford a static\narray that large. We also need to make sure the compiler doesn’t overflow that\nlimit.

\n
    if (upvalue->index == index && upvalue->isLocal == isLocal) {\n      return i;\n    }\n  }\n\n
compiler.c
\nin addUpvalue()
\n
  if (upvalueCount == UINT8_COUNT) {\n    error("Too many closure variables in function.");\n    return 0;\n  }\n\n
  compiler->upvalues[upvalueCount].isLocal = isLocal;\n
\n
compiler.c, in addUpvalue()
\n\n

Finally, the Upvalue struct type itself.

\n
compiler.c
\nadd after struct Local
\n
typedef struct {\n  uint8_t index;\n  bool isLocal;\n} Upvalue;\n
\n
compiler.c, add after struct Local
\n\n

The index field stores which local slot the upvalue is capturing. The\nisLocal field deserves its own section, which we’ll get to next.

\n

25 . 2 . 2Flattening upvalues

\n

In the example I showed before, the closure is accessing a variable declared in\nthe immediately enclosing function. Lox also supports accessing local variables\ndeclared in any enclosing scope, as in:

\n
fun outer() {\n  var x = 1;\n  fun middle() {\n    fun inner() {\n      print x;\n    }\n  }\n}\n
\n

Here, we’re accessing x in inner(). That variable is defined not in\nmiddle(), but all the way out in outer(). We need to handle cases like this\ntoo. You might think that this isn’t much harder since the variable will\nsimply be somewhere farther down on the stack. But consider this devious example:

\n\n
fun outer() {\n  var x = "value";\n  fun middle() {\n    fun inner() {\n      print x;\n    }\n\n    print "create inner closure";\n    return inner;\n  }\n\n  print "return from outer";\n  return middle;\n}\n\nvar mid = outer();\nvar in = mid();\nin();\n
\n

When you run this, it should print:

\n
return from outer\ncreate inner closure\nvalue\n
\n

I know, it’s convoluted. The important part is that outer()—where x is\ndeclared—returns and pops all of its variables off the stack before the\ndeclaration of inner() executes. So, at the point in time that we create the\nclosure for inner(), x is already off the stack.

\n

Here, I traced out the execution flow for you:

\"Tracing\n

See how x is popped ① before it is captured ② and then later\naccessed ③? We really have two problems:

\n
    \n
  1. \n

    We need to resolve local variables that are declared in surrounding\nfunctions beyond the immediately enclosing one.

    \n
    2. \n
  2. \n

    We need to be able to capture variables that have already left the stack.

    \n
    3. \n
\n

Fortunately, we’re in the middle of adding upvalues to the VM, and upvalues are\nexplicitly designed for tracking variables that have escaped the stack. So, in a\nclever bit of self-reference, we can use upvalues to allow upvalues to capture\nvariables declared outside of the immediately surrounding function.

\n

The solution is to allow a closure to capture either a local variable or an\nexisting upvalue in the immediately enclosing function. If a deeply nested\nfunction references a local variable declared several hops away, we’ll thread it\nthrough all of the intermediate functions by having each function capture an\nupvalue for the next function to grab.

\"An\n

In the above example, middle() captures the local variable x in the\nimmediately enclosing function outer() and stores it in its own upvalue. It\ndoes this even though middle() itself doesn’t reference x. Then, when the\ndeclaration of inner() executes, its closure grabs the upvalue from the\nObjClosure for middle() that captured x. A function captures—either a\nlocal or upvalue—only from the immediately surrounding function, which is\nguaranteed to still be around at the point that the inner function declaration\nexecutes.

\n

In order to implement this, resolveUpvalue() becomes recursive.

\n
  if (local != -1) {\n    return addUpvalue(compiler, (uint8_t)local, true);\n  }\n\n
compiler.c
\nin resolveUpvalue()
\n
  int upvalue = resolveUpvalue(compiler->enclosing, name);\n  if (upvalue != -1) {\n    return addUpvalue(compiler, (uint8_t)upvalue, false);\n  }\n\n
  return -1;\n
\n
compiler.c, in resolveUpvalue()
\n\n

It’s only another three lines of code, but I found this function really\nchallenging to get right the first time. This in spite of the fact that I wasn’t\ninventing anything new, just porting the concept over from Lua. Most recursive\nfunctions either do all their work before the recursive call (a pre-order\ntraversal, or “on the way down”), or they do all the work after the recursive\ncall (a post-order traversal, or “on the way back up”). This function does\nboth. The recursive call is right in the middle.

\n

We’ll walk through it slowly. First, we look for a matching local variable in\nthe enclosing function. If we find one, we capture that local and return. That’s\nthe base case.

\n\n

Otherwise, we look for a local variable beyond the immediately enclosing\nfunction. We do that by recursively calling resolveUpvalue() on the\nenclosing compiler, not the current one. This series of resolveUpvalue()\ncalls works its way along the chain of nested compilers until it hits one of\nthe base cases—either it finds an actual local variable to capture or it\nruns out of compilers.

\n

When a local variable is found, the most deeply nested\ncall to resolveUpvalue() captures it and returns the upvalue index. That\nreturns to the next call for the inner function declaration. That call captures\nthe upvalue from the surrounding function, and so on. As each nested call to\nresolveUpvalue() returns, we drill back down into the innermost function\ndeclaration where the identifier we are resolving appears. At each step along\nthe way, we add an upvalue to the intervening function and pass the resulting\nupvalue index down to the next call.

\n\n

It might help to walk through the original example when resolving x:

\"Tracing\n

Note that the new call to addUpvalue() passes false for the isLocal\nparameter. Now you see that that flag controls whether the closure captures a\nlocal variable or an upvalue from the surrounding function.

\n

By the time the compiler reaches the end of a function declaration, every\nvariable reference has been resolved as either a local, an upvalue, or a global.\nEach upvalue may in turn capture a local variable from the surrounding function,\nor an upvalue in the case of transitive closures. We finally have enough data to\nemit bytecode which creates a closure at runtime that captures all of the\ncorrect variables.

\n
  emitBytes(OP_CLOSURE, makeConstant(OBJ_VAL(function)));\n
compiler.c
\nin function()
\n
\n\n  for (int i = 0; i < function->upvalueCount; i++) {\n    emitByte(compiler.upvalues[i].isLocal ? 1 : 0);\n    emitByte(compiler.upvalues[i].index);\n  }\n
}\n
\n
compiler.c, in function()
\n\n

The OP_CLOSURE instruction is unique in that it has a variably sized encoding.\nFor each upvalue the closure captures, there are two single-byte operands. Each\npair of operands specifies what that upvalue captures. If the first byte is one,\nit captures a local variable in the enclosing function. If zero, it captures one\nof the function’s upvalues. The next byte is the local slot or upvalue index to\ncapture.

\n

This odd encoding means we need some bespoke support in the disassembly code\nfor OP_CLOSURE.

\n
      printf("\\n");\n
debug.c
\nin disassembleInstruction()
\n
\n\n      ObjFunction* function = AS_FUNCTION(\n          chunk->constants.values[constant]);\n      for (int j = 0; j < function->upvalueCount; j++) {\n        int isLocal = chunk->code[offset++];\n        int index = chunk->code[offset++];\n        printf("%04d      |                     %s %d\\n",\n               offset - 2, isLocal ? "local" : "upvalue", index);\n      }\n\n
      return offset;\n
\n
debug.c, in disassembleInstruction()
\n\n

For example, take this script:

\n
fun outer() {\n  var a = 1;\n  var b = 2;\n  fun middle() {\n    var c = 3;\n    var d = 4;\n    fun inner() {\n      print a + c + b + d;\n    }\n  }\n}\n
\n

If we disassemble the instruction that creates the closure for inner(), it\nprints this:

\n
0004    9 OP_CLOSURE          2 <fn inner>\n0006      |                     upvalue 0\n0008      |                     local 1\n0010      |                     upvalue 1\n0012      |                     local 2\n
\n

We have two other, simpler instructions to add disassembler support for.

\n
    case OP_SET_GLOBAL:\n      return constantInstruction("OP_SET_GLOBAL", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_GET_UPVALUE:\n      return byteInstruction("OP_GET_UPVALUE", chunk, offset);\n    case OP_SET_UPVALUE:\n      return byteInstruction("OP_SET_UPVALUE", chunk, offset);\n
    case OP_EQUAL:\n
\n
debug.c, in disassembleInstruction()
\n\n

These both have a single-byte operand, so there’s nothing exciting going on. We\ndo need to add an include so the debug module can get to AS_FUNCTION().

\n
#include "debug.h"\n
debug.c
\n
#include "object.h"\n
#include "value.h"\n
\n
debug.c
\n\n

With that, our compiler is where we want it. For each function declaration, it\noutputs an OP_CLOSURE instruction followed by a series of operand byte pairs\nfor each upvalue it needs to capture at runtime. It’s time to hop over to that\nside of the VM and get things running.

\n

25 . 3Upvalue Objects

\n

Each OP_CLOSURE instruction is now followed by the series of bytes that\nspecify the upvalues the ObjClosure should own. Before we process those\noperands, we need a runtime representation for upvalues.

\n
object.h
\nadd after struct ObjString
\n
typedef struct ObjUpvalue {\n  Obj obj;\n  Value* location;\n} ObjUpvalue;\n
\n
object.h, add after struct ObjString
\n\n

We know upvalues must manage closed-over variables that no longer live on the\nstack, which implies some amount of dynamic allocation. The easiest way to do\nthat in our VM is by building on the object system we already have. That way,\nwhen we implement a garbage collector in the next chapter, the GC can\nmanage memory for upvalues too.

\n

Thus, our runtime upvalue structure is an ObjUpvalue with the typical Obj header\nfield. Following that is a location field that points to the closed-over\nvariable. Note that this is a pointer to a Value, not a Value itself. It’s a\nreference to a variable, not a value. This is important because it means\nthat when we assign to the variable the upvalue captures, we’re assigning to the\nactual variable, not a copy. For example:

\n
fun outer() {\n  var x = "before";\n  fun inner() {\n    x = "assigned";\n  }\n  inner();\n  print x;\n}\nouter();\n
\n

This program should print “assigned” even though the closure assigns to x and\nthe surrounding function accesses it.

\n

Because upvalues are objects, we’ve got all the usual object machinery, starting\nwith a constructor-like function:

\n
ObjString* copyString(const char* chars, int length);\n
object.h
\nadd after copyString()
\n
ObjUpvalue* newUpvalue(Value* slot);\n
void printObject(Value value);\n
\n
object.h, add after copyString()
\n\n

It takes the address of the slot where the closed-over variable lives. Here is\nthe implementation:

\n
object.c
\nadd after copyString()
\n
ObjUpvalue* newUpvalue(Value* slot) {\n  ObjUpvalue* upvalue = ALLOCATE_OBJ(ObjUpvalue, OBJ_UPVALUE);\n  upvalue->location = slot;\n  return upvalue;\n}\n
\n
object.c, add after copyString()
\n\n

We simply initialize the object and store the pointer. That requires a new\nobject type.

\n
  OBJ_STRING,\n
object.h
\nin enum ObjType
\n
  OBJ_UPVALUE\n
} ObjType;\n
\n
object.h, in enum ObjType
\n\n

And on the back side, a destructor-like function:

\n
      FREE(ObjString, object);\n      break;\n    }\n
memory.c
\nin freeObject()
\n
    case OBJ_UPVALUE:\n      FREE(ObjUpvalue, object);\n      break;\n
  }\n
\n
memory.c, in freeObject()
\n\n

Multiple closures can close over the same variable, so ObjUpvalue does not own\nthe variable it references. Thus, the only thing to free is the ObjUpvalue\nitself.

\n

And, finally, to print:

\n
    case OBJ_STRING:\n      printf("%s", AS_CSTRING(value));\n      break;\n
object.c
\nin printObject()
\n
    case OBJ_UPVALUE:\n      printf("upvalue");\n      break;\n
  }\n
\n
object.c, in printObject()
\n\n

Printing isn’t useful to end users. Upvalues are objects only so that we can\ntake advantage of the VM’s memory management. They aren’t first-class values\nthat a Lox user can directly access in a program. So this code will never\nactually execute . . . but it keeps the compiler from yelling at us about an\nunhandled switch case, so here we are.

\n

25 . 3 . 1Upvalues in closures

\n

When I first introduced upvalues, I said each closure has an array of them.\nWe’ve finally worked our way back to implementing that.

\n
  ObjFunction* function;\n
object.h
\nin struct ObjClosure
\n
  ObjUpvalue** upvalues;\n  int upvalueCount;\n
} ObjClosure;\n
\n
object.h, in struct ObjClosure
\n\n

Different closures may have different numbers of\nupvalues, so we need a dynamic array. The upvalues themselves are dynamically\nallocated too, so we end up with a double pointer—a pointer to a dynamically\nallocated array of pointers to upvalues. We also store the number of elements in\nthe array.

\n\n

When we create an ObjClosure, we allocate an upvalue array of the proper size,\nwhich we determined at compile time and stored in the ObjFunction.

\n
ObjClosure* newClosure(ObjFunction* function) {\n
object.c
\nin newClosure()
\n
  ObjUpvalue** upvalues = ALLOCATE(ObjUpvalue*,\n                                   function->upvalueCount);\n  for (int i = 0; i < function->upvalueCount; i++) {\n    upvalues[i] = NULL;\n  }\n\n
  ObjClosure* closure = ALLOCATE_OBJ(ObjClosure, OBJ_CLOSURE);\n
\n
object.c, in newClosure()
\n\n

Before creating the closure object itself, we allocate the array of upvalues and\ninitialize them all to NULL. This weird ceremony around memory is a careful\ndance to please the (forthcoming) garbage collection deities. It ensures the\nmemory manager never sees uninitialized memory.

\n

Then we store the array in the new closure, as well as copy the count over from\nthe ObjFunction.

\n
  closure->function = function;\n
object.c
\nin newClosure()
\n
  closure->upvalues = upvalues;\n  closure->upvalueCount = function->upvalueCount;\n
  return closure;\n
\n
object.c, in newClosure()
\n\n

When we free an ObjClosure, we also free the upvalue array.

\n
    case OBJ_CLOSURE: {\n
memory.c
\nin freeObject()
\n
      ObjClosure* closure = (ObjClosure*)object;\n      FREE_ARRAY(ObjUpvalue*, closure->upvalues,\n                 closure->upvalueCount);\n
      FREE(ObjClosure, object);\n
\n
memory.c, in freeObject()
\n\n

ObjClosure does not own the ObjUpvalue objects themselves, but it does own the\narray containing pointers to those upvalues.

\n

We fill the upvalue array over in the interpreter when it creates a closure.\nThis is where we walk through all of the operands after OP_CLOSURE to see what\nkind of upvalue each slot captures.

\n
        push(OBJ_VAL(closure));\n
vm.c
\nin run()
\n
        for (int i = 0; i < closure->upvalueCount; i++) {\n          uint8_t isLocal = READ_BYTE();\n          uint8_t index = READ_BYTE();\n          if (isLocal) {\n            closure->upvalues[i] =\n                captureUpvalue(frame->slots + index);\n          } else {\n            closure->upvalues[i] = frame->closure->upvalues[index];\n          }\n        }\n
        break;\n
\n
vm.c, in run()
\n\n

This code is the magic moment when a closure comes to life. We iterate over each\nupvalue the closure expects. For each one, we read a pair of operand bytes. If\nthe upvalue closes over a local variable in the enclosing function, we let\ncaptureUpvalue() do the work.

\n

Otherwise, we capture an upvalue from the surrounding function. An OP_CLOSURE\ninstruction is emitted at the end of a function declaration. At the moment that\nwe are executing that declaration, the current function is the surrounding\none. That means the current function’s closure is stored in the CallFrame at the\ntop of the callstack. So, to grab an upvalue from the enclosing function, we can\nread it right from the frame local variable, which caches a reference to that\nCallFrame.

\n

Closing over a local variable is more interesting. Most of the work happens in a\nseparate function, but first we calculate the argument to pass to it. We need to\ngrab a pointer to the captured local’s slot in the surrounding function’s stack\nwindow. That window begins at frame->slots, which points to slot zero. Adding\nindex offsets that to the local slot we want to capture. We pass that pointer\nhere:

\n
vm.c
\nadd after callValue()
\n
static ObjUpvalue* captureUpvalue(Value* local) {\n  ObjUpvalue* createdUpvalue = newUpvalue(local);\n  return createdUpvalue;\n}\n
\n
vm.c, add after callValue()
\n\n

This seems a little silly. All it does is create a new ObjUpvalue that captures\nthe given stack slot and returns it. Did we need a separate function for this?\nWell, no, not yet. But you know we are going to end up sticking more code in\nhere.

\n

First, let’s wrap up what we’re working on. Back in the interpreter code for\nhandling OP_CLOSURE, we eventually finish iterating through the upvalue\narray and initialize each one. When that completes, we have a new closure with\nan array full of upvalues pointing to variables.

\n

With that in hand, we can implement the instructions that work with those\nupvalues.

\n
      }\n
vm.c
\nin run()
\n
      case OP_GET_UPVALUE: {\n        uint8_t slot = READ_BYTE();\n        push(*frame->closure->upvalues[slot]->location);\n        break;\n      }\n
      case OP_EQUAL: {\n
\n
vm.c, in run()
\n\n

The operand is the index into the current function’s upvalue array. So we simply\nlook up the corresponding upvalue and dereference its location pointer to read\nthe value in that slot. Setting a variable is similar.

\n
      }\n
vm.c
\nin run()
\n
      case OP_SET_UPVALUE: {\n        uint8_t slot = READ_BYTE();\n        *frame->closure->upvalues[slot]->location = peek(0);\n        break;\n      }\n
      case OP_EQUAL: {\n
\n
vm.c, in run()
\n\n

We take the value on top of the stack and store it\ninto the slot pointed to by the chosen upvalue. Just as with the instructions\nfor local variables, it’s important that these instructions are fast. User\nprograms are constantly reading and writing variables, so if that’s slow,\neverything is slow. And, as usual, the way we make them fast is by keeping them\nsimple. These two new instructions are pretty good: no control flow, no complex\narithmetic, just a couple of pointer indirections and a push().

\n\n

This is a milestone. As long as all of the variables remain on the stack, we\nhave working closures. Try this:

\n
fun outer() {\n  var x = "outside";\n  fun inner() {\n    print x;\n  }\n  inner();\n}\nouter();\n
\n

Run this, and it correctly prints “outside”.

\n

25 . 4Closed Upvalues

\n

Of course, a key feature of closures is that they hold on to the variable as\nlong as needed, even after the function that declares the variable has returned.\nHere’s another example that should work:

\n
fun outer() {\n  var x = "outside";\n  fun inner() {\n    print x;\n  }\n\n  return inner;\n}\n\nvar closure = outer();\nclosure();\n
\n

But if you run it right now . . . who knows what it does? At runtime, it will end\nup reading from a stack slot that no longer contains the closed-over variable.\nLike I’ve mentioned a few times, the crux of the issue is that variables in\nclosures don’t have stack semantics. That means we’ve got to hoist them off the\nstack when the function where they were declared returns. This final section of\nthe chapter does that.

\n

25 . 4 . 1Values and variables

\n

Before we get to writing code, I want to dig into an important semantic point.\nDoes a closure close over a value or a variable? This isn’t purely an academic question. I’m not just splitting hairs.\nConsider:

\n\n
var globalSet;\nvar globalGet;\n\nfun main() {\n  var a = "initial";\n\n  fun set() { a = "updated"; }\n  fun get() { print a; }\n\n  globalSet = set;\n  globalGet = get;\n}\n\nmain();\nglobalSet();\nglobalGet();\n
\n

The outer main() function creates two closures and stores them in global variables so that they outlive the execution of\nmain() itself. Both of those closures capture the same variable. The first\nclosure assigns a new value to it and the second closure reads the variable.

\n\n

What does the call to globalGet() print? If closures capture values then\neach closure gets its own copy of a with the value that a had at the point\nin time that the closure’s function declaration executed. The call to\nglobalSet() will modify set()’s copy of a, but get()’s copy will be\nunaffected. Thus, the call to globalGet() will print “initial”.

\n

If closures close over variables, then get() and set() will both capture—reference—the same mutable variable. When set() changes a, it changes\nthe same a that get() reads from. There is only one a. That, in turn,\nimplies the call to globalGet() will print “updated”.

\n

Which is it? The answer for Lox and most other languages I know with closures is\nthe latter. Closures capture variables. You can think of them as capturing the\nplace the value lives. This is important to keep in mind as we deal with\nclosed-over variables that are no longer on the stack. When a variable moves to\nthe heap, we need to ensure that all closures capturing that variable retain a\nreference to its one new location. That way, when the variable is mutated, all\nclosures see the change.

\n

25 . 4 . 2Closing upvalues

\n

We know that local variables always start out on the stack. This is faster, and\nlets our single-pass compiler emit code before it discovers the variable has\nbeen captured. We also know that closed-over variables need to move to the heap\nif the closure outlives the function where the captured variable is declared.

\n

Following Lua, we’ll use open upvalue to refer to an upvalue that points to\na local variable still on the stack. When a variable moves to the heap, we are\nclosing the upvalue and the result is, naturally, a closed upvalue. The\ntwo questions we need to answer are:

\n
    \n
  1. \n

    Where on the heap does the closed-over variable go?

    \n
    2. \n
  2. \n

    When do we close the upvalue?

    \n
    3. \n
\n

The answer to the first question is easy. We already have a convenient object on\nthe heap that represents a reference to a variable—ObjUpvalue itself. The\nclosed-over variable will move into a new field right inside the ObjUpvalue\nstruct. That way we don’t need to do any additional heap allocation to close an\nupvalue.

\n

The second question is straightforward too. As long as the variable is on the\nstack, there may be code that refers to it there, and that code must work\ncorrectly. So the logical time to hoist the variable to the heap is as late as\npossible. If we move the local variable right when it goes out of scope, we are\ncertain that no code after that point will try to access it from the stack.\nAfter the variable is out of scope, the compiler will\nhave reported an error if any code tried to use it.

\n\n

The compiler already emits an OP_POP instruction when a local variable goes\nout of scope. If a variable is captured by a closure, we will instead emit a\ndifferent instruction to hoist that variable out of the stack and into its\ncorresponding upvalue. To do that, the compiler needs to know which locals are closed over.

\n\n

The compiler already maintains an array of Upvalue structs for each local\nvariable in the function to track exactly that state. That array is good for\nanswering “Which variables does this closure use?” But it’s poorly suited for\nanswering, “Does any function capture this local variable?” In particular,\nonce the Compiler for some closure has finished, the Compiler for the enclosing\nfunction whose variable has been captured no longer has access to any of the\nupvalue state.

\n

In other words, the compiler maintains pointers from upvalues to the locals they\ncapture, but not in the other direction. So we first need to add some extra\ntracking inside the existing Local struct so that we can tell if a given local\nis captured by a closure.

\n
  int depth;\n
compiler.c
\nin struct Local
\n
  bool isCaptured;\n
} Local;\n
\n
compiler.c, in struct Local
\n\n

This field is true if the local is captured by any later nested function\ndeclaration. Initially, all locals are not captured.

\n
  local->depth = -1;\n
compiler.c
\nin addLocal()
\n
  local->isCaptured = false;\n
}\n
\n
compiler.c, in addLocal()
\n\n

Likewise, the special “slot zero local” that the\ncompiler implicitly declares is not captured.

\n\n
  local->depth = 0;\n
compiler.c
\nin initCompiler()
\n
  local->isCaptured = false;\n
  local->name.start = "";\n
\n
compiler.c, in initCompiler()
\n\n

When resolving an identifier, if we end up creating an upvalue for a local\nvariable, we mark it as captured.

\n
  if (local != -1) {\n
compiler.c
\nin resolveUpvalue()
\n
    compiler->enclosing->locals[local].isCaptured = true;\n
    return addUpvalue(compiler, (uint8_t)local, true);\n
\n
compiler.c, in resolveUpvalue()
\n\n

Now, at the end of a block scope when the compiler emits code to free the stack\nslots for the locals, we can tell which ones need to get hoisted onto the heap.\nWe’ll use a new instruction for that.

\n
  while (current->localCount > 0 &&\n         current->locals[current->localCount - 1].depth >\n            current->scopeDepth) {\n
compiler.c
\nin endScope()
\nreplace 1 line
\n
    if (current->locals[current->localCount - 1].isCaptured) {\n      emitByte(OP_CLOSE_UPVALUE);\n    } else {\n      emitByte(OP_POP);\n    }\n
    current->localCount--;\n  }\n
\n
compiler.c, in endScope(), replace 1 line
\n\n

The instruction requires no operand. We know that the variable will always be\nright on top of the stack at the point that this instruction executes. We\ndeclare the instruction.

\n
  OP_CLOSURE,\n
chunk.h
\nin enum OpCode
\n
  OP_CLOSE_UPVALUE,\n
  OP_RETURN,\n
\n
chunk.h, in enum OpCode
\n\n

And add trivial disassembler support for it:

\n
    }\n
debug.c
\nin disassembleInstruction()
\n
    case OP_CLOSE_UPVALUE:\n      return simpleInstruction("OP_CLOSE_UPVALUE", offset);\n
    case OP_RETURN:\n
\n
debug.c, in disassembleInstruction()
\n\n

Excellent. Now the generated bytecode tells the runtime exactly when each\ncaptured local variable must move to the heap. Better, it does so only for the\nlocals that are used by a closure and need this special treatment. This aligns\nwith our general performance goal that we want users to pay only for\nfunctionality that they use. Variables that aren’t used by closures live and die\nentirely on the stack just as they did before.

\n

25 . 4 . 3Tracking open upvalues

\n

Let’s move over to the runtime side. Before we can interpret OP_CLOSE_UPVALUE\ninstructions, we have an issue to resolve. Earlier, when I talked about whether\nclosures capture variables or values, I said it was important that if multiple\nclosures access the same variable that they end up with a reference to the\nexact same storage location in memory. That way if one closure writes to the\nvariable, the other closure sees the change.

\n

Right now, if two closures capture the same local\nvariable, the VM creates a separate Upvalue for each one. The necessary sharing\nis missing. When we move the variable off the stack, if we move it into only one\nof the upvalues, the other upvalue will have an orphaned value.

\n\n

To fix that, whenever the VM needs an upvalue that captures a particular local\nvariable slot, we will first search for an existing upvalue pointing to that\nslot. If found, we reuse that. The challenge is that all of the previously\ncreated upvalues are squirreled away inside the upvalue arrays of the various\nclosures. Those closures could be anywhere in the VM’s memory.

\n

The first step is to give the VM its own list of all open upvalues that point to\nvariables still on the stack. Searching a list each time the VM needs an upvalue\nsounds like it might be slow, but in practice, it’s not bad. The number of\nvariables on the stack that actually get closed over tends to be small. And\nfunction declarations that create closures are rarely\non performance critical execution paths in the user’s program.

\n\n

Even better, we can order the list of open upvalues by the stack slot index they\npoint to. The common case is that a slot has not already been captured—sharing variables between closures is uncommon—and closures tend to capture\nlocals near the top of the stack. If we store the open upvalue array in stack\nslot order, as soon as we step past the slot where the local we’re capturing\nlives, we know it won’t be found. When that local is near the top of the stack,\nwe can exit the loop pretty early.

\n

Maintaining a sorted list requires inserting elements in the middle efficiently.\nThat suggests using a linked list instead of a dynamic array. Since we defined\nthe ObjUpvalue struct ourselves, the easiest implementation is an intrusive list\nthat puts the next pointer right inside the ObjUpvalue struct itself.

\n
  Value* location;\n
object.h
\nin struct ObjUpvalue
\n
  struct ObjUpvalue* next;\n
} ObjUpvalue;\n
\n
object.h, in struct ObjUpvalue
\n\n

When we allocate an upvalue, it is not attached to any list yet so the link is\nNULL.

\n
  upvalue->location = slot;\n
object.c
\nin newUpvalue()
\n
  upvalue->next = NULL;\n
  return upvalue;\n
\n
object.c, in newUpvalue()
\n\n

The VM owns the list, so the head pointer goes right inside the main VM struct.

\n
  Table strings;\n
vm.h
\nin struct VM
\n
  ObjUpvalue* openUpvalues;\n
  Obj* objects;\n
\n
vm.h, in struct VM
\n\n

The list starts out empty.

\n
  vm.frameCount = 0;\n
vm.c
\nin resetStack()
\n
  vm.openUpvalues = NULL;\n
}\n
\n
vm.c, in resetStack()
\n\n

Starting with the first upvalue pointed to by the VM, each open upvalue points\nto the next open upvalue that references a local variable farther down the\nstack. This script, for example,

\n
{\n  var a = 1;\n  fun f() {\n    print a;\n  }\n  var b = 2;\n  fun g() {\n    print b;\n  }\n  var c = 3;\n  fun h() {\n    print c;\n  }\n}\n
\n

should produce a series of linked upvalues like so:

\"Three\n

Whenever we close over a local variable, before creating a new upvalue, we look\nfor an existing one in the list.

\n
static ObjUpvalue* captureUpvalue(Value* local) {\n
vm.c
\nin captureUpvalue()
\n
  ObjUpvalue* prevUpvalue = NULL;\n  ObjUpvalue* upvalue = vm.openUpvalues;\n  while (upvalue != NULL && upvalue->location > local) {\n    prevUpvalue = upvalue;\n    upvalue = upvalue->next;\n  }\n\n  if (upvalue != NULL && upvalue->location == local) {\n    return upvalue;\n  }\n\n
  ObjUpvalue* createdUpvalue = newUpvalue(local);\n
\n
vm.c, in captureUpvalue()
\n\n

We start at the head of the list, which is the upvalue\nclosest to the top of the stack. We walk through the list, using a little\npointer comparison to iterate past every upvalue pointing to slots above the one\nwe’re looking for. While we do that, we keep track of the preceding upvalue on\nthe list. We’ll need to update that node’s next pointer if we end up inserting\na node after it.

\n\n

There are three reasons we can exit the loop:

\n
    \n
  1. \n

    The local slot we stopped at is the slot we’re looking for. We found\nan existing upvalue capturing the variable, so we reuse that upvalue.

    \n
    2. \n
  2. \n

    We ran out of upvalues to search. When upvalue is NULL, it means\nevery open upvalue in the list points to locals above the slot we’re looking\nfor, or (more likely) the upvalue list is empty. Either way, we didn’t find\nan upvalue for our slot.

    \n
    3. \n
  3. \n

    We found an upvalue whose local slot is below the one we’re looking\nfor. Since the list is sorted, that means we’ve gone past the slot we are\nclosing over, and thus there must not be an existing upvalue for it.

    \n
    4. \n
\n

In the first case, we’re done and we’ve returned. Otherwise, we create a new\nupvalue for our local slot and insert it into the list at the right location.

\n
  ObjUpvalue* createdUpvalue = newUpvalue(local);\n
vm.c
\nin captureUpvalue()
\n
  createdUpvalue->next = upvalue;\n\n  if (prevUpvalue == NULL) {\n    vm.openUpvalues = createdUpvalue;\n  } else {\n    prevUpvalue->next = createdUpvalue;\n  }\n\n
  return createdUpvalue;\n
\n
vm.c, in captureUpvalue()
\n\n

The current incarnation of this function already creates the upvalue, so we only\nneed to add code to insert the upvalue into the list. We exited the list\ntraversal by either going past the end of the list, or by stopping on the first\nupvalue whose stack slot is below the one we’re looking for. In either case,\nthat means we need to insert the new upvalue before the object pointed at by\nupvalue (which may be NULL if we hit the end of the list).

\n

As you may have learned in Data Structures 101, to insert a node into a linked\nlist, you set the next pointer of the previous node to point to your new one.\nWe have been conveniently keeping track of that preceding node as we walked the\nlist. We also need to handle the special case where\nwe are inserting a new upvalue at the head of the list, in which case the “next”\npointer is the VM’s head pointer.

\n\n

With this updated function, the VM now ensures that there is only ever a single\nObjUpvalue for any given local slot. If two closures capture the same variable,\nthey will get the same upvalue. We’re ready to move those upvalues off the\nstack now.

\n

25 . 4 . 4Closing upvalues at runtime

\n

The compiler helpfully emits an OP_CLOSE_UPVALUE instruction to tell the VM\nexactly when a local variable should be hoisted onto the heap. Executing that\ninstruction is the interpreter’s responsibility.

\n
      }\n
vm.c
\nin run()
\n
      case OP_CLOSE_UPVALUE:\n        closeUpvalues(vm.stackTop - 1);\n        pop();\n        break;\n
      case OP_RETURN: {\n
\n
vm.c, in run()
\n\n

When we reach the instruction, the variable we are hoisting is right on top of\nthe stack. We call a helper function, passing the address of that stack slot.\nThat function is responsible for closing the upvalue and moving the local from\nthe stack to the heap. After that, the VM is free to discard the stack slot,\nwhich it does by calling pop().

\n

The fun stuff happens here:

\n
vm.c
\nadd after captureUpvalue()
\n
static void closeUpvalues(Value* last) {\n  while (vm.openUpvalues != NULL &&\n         vm.openUpvalues->location >= last) {\n    ObjUpvalue* upvalue = vm.openUpvalues;\n    upvalue->closed = *upvalue->location;\n    upvalue->location = &upvalue->closed;\n    vm.openUpvalues = upvalue->next;\n  }\n}\n
\n
vm.c, add after captureUpvalue()
\n\n

This function takes a pointer to a stack slot. It closes every open upvalue it\ncan find that points to that slot or any slot above it on the stack. Right now,\nwe pass a pointer only to the top slot on the stack, so the “or above it” part\ndoesn’t come into play, but it will soon.

\n

To do this, we walk the VM’s list of open upvalues, again from top to bottom. If\nan upvalue’s location points into the range of slots we’re closing, we close the\nupvalue. Otherwise, once we reach an upvalue outside of the range, we know the\nrest will be too, so we stop iterating.

\n

The way an upvalue gets closed is pretty cool. First,\nwe copy the variable’s value into the closed field in the ObjUpvalue. That’s\nwhere closed-over variables live on the heap. The OP_GET_UPVALUE and\nOP_SET_UPVALUE instructions need to look for the variable there after it’s\nbeen moved. We could add some conditional logic in the interpreter code for\nthose instructions to check some flag for whether the upvalue is open or closed.

\n

But there is already a level of indirection in play—those instructions\ndereference the location pointer to get to the variable’s value. When the\nvariable moves from the stack to the closed field, we simply update that\nlocation to the address of the ObjUpvalue’s own closed field.

\n\"Moving\n

We don’t need to change how OP_GET_UPVALUE and OP_SET_UPVALUE are\ninterpreted at all. That keeps them simple, which in turn keeps them fast. We do\nneed to add the new field to ObjUpvalue, though.

\n
  Value* location;\n
object.h
\nin struct ObjUpvalue
\n
  Value closed;\n
  struct ObjUpvalue* next;\n
\n
object.h, in struct ObjUpvalue
\n\n

And we should zero it out when we create an ObjUpvalue so there’s no\nuninitialized memory floating around.

\n
  ObjUpvalue* upvalue = ALLOCATE_OBJ(ObjUpvalue, OBJ_UPVALUE);\n
object.c
\nin newUpvalue()
\n
  upvalue->closed = NIL_VAL;\n
  upvalue->location = slot;\n
\n
object.c, in newUpvalue()
\n\n

Whenever the compiler reaches the end of a block, it discards all local\nvariables in that block and emits an OP_CLOSE_UPVALUE for each local variable\nthat was closed over. The compiler does not emit any\ninstructions at the end of the outermost block scope that defines a function\nbody. That scope contains the function’s parameters and any locals declared\nimmediately inside the function. Those need to get closed too.

\n\n

This is the reason closeUpvalues() accepts a pointer to a stack slot. When a\nfunction returns, we call that same helper and pass in the first stack slot\nowned by the function.

\n
        Value result = pop();\n
vm.c
\nin run()
\n
        closeUpvalues(frame->slots);\n
        vm.frameCount--;\n
\n
vm.c, in run()
\n\n

By passing the first slot in the function’s stack window, we close every\nremaining open upvalue owned by the returning function. And with that, we now\nhave a fully functioning closure implementation. Closed-over variables live as\nlong as they are needed by the functions that capture them.

\n

This was a lot of work! In jlox, closures fell out naturally from our\nenvironment representation. In clox, we had to add a lot of code—new bytecode\ninstructions, more data structures in the compiler, and new runtime objects. The\nVM very much treats variables in closures as different from other variables.

\n

There is a rationale for that. In terms of implementation complexity, jlox gave\nus closures “for free”. But in terms of performance, jlox’s closures are\nanything but. By allocating all environments on the heap, jlox pays a\nsignificant performance price for all local variables, even the majority which\nare never captured by closures.

\n

With clox, we have a more complex system, but that allows us to tailor the\nimplementation to fit the two use patterns we observe for local variables. For\nmost variables which do have stack semantics, we allocate them entirely on the\nstack which is simple and fast. Then, for the few local variables where that\ndoesn’t work, we have a second slower path we can opt in to as needed.

\n

Fortunately, users don’t perceive the complexity. From their perspective, local\nvariables in Lox are simple and uniform. The language itself is as simple as\njlox’s implementation. But under the hood, clox is watching what the user does\nand optimizing for their specific uses. As your language implementations grow in\nsophistication, you’ll find yourself doing this more. A large fraction of\n“optimization” is about adding special case code that detects certain uses and\nprovides a custom-built, faster path for code that fits that pattern.

\n

We have lexical scoping fully working in clox now, which is a major milestone.\nAnd, now that we have functions and variables with complex lifetimes, we also\nhave a lot of objects floating around in clox’s heap, with a web of pointers\nstringing them together. The next step is figuring out how to manage that\nmemory so that we can free some of those objects when they’re no longer needed.

\n
\n

Challenges

\n
    \n
  1. \n

    Wrapping every ObjFunction in an ObjClosure introduces a level of\nindirection that has a performance cost. That cost isn’t necessary for\nfunctions that do not close over any variables, but it does let the runtime\ntreat all calls uniformly.

    \n

    Change clox to only wrap functions in ObjClosures that need upvalues. How\ndoes the code complexity and performance compare to always wrapping\nfunctions? Take care to benchmark programs that do and do not use closures.\nHow should you weight the importance of each benchmark? If one gets slower\nand one faster, how do you decide what trade-off to make to choose an\nimplementation strategy?

    \n
    2. \n
  2. \n

    Read the design note below. I’ll wait. Now, how do you think Lox should\nbehave? Change the implementation to create a new variable for each loop\niteration.

    \n
    3. \n
  3. \n

    A famous koan teaches us that “objects are a poor man’s closure”\n(and vice versa). Our VM doesn’t support objects yet, but now that we have\nclosures we can approximate them. Using closures, write a Lox program that\nmodels two-dimensional vector “objects”. It should:

    \n
      \n
    • \n

      Define a “constructor” function to create a new vector with the given\nx and y coordinates.

      \n
      • \n
    • \n

      Provide “methods” to access the x and y coordinates of values\nreturned from that constructor.

      \n
      • \n
    • \n

      Define an addition “method” that adds two vectors and produces a third.

      \n
      • \n
    \n
    4. \n
\n
\n
\n

Design Note: Closing Over the Loop Variable

\n

Closures capture variables. When two closures capture the same variable, they\nshare a reference to the same underlying storage location. This fact is visible\nwhen new values are assigned to the variable. Obviously, if two closures capture\ndifferent variables, there is no sharing.

\n
var globalOne;\nvar globalTwo;\n\nfun main() {\n  {\n    var a = "one";\n    fun one() {\n      print a;\n    }\n    globalOne = one;\n  }\n\n  {\n    var a = "two";\n    fun two() {\n      print a;\n    }\n    globalTwo = two;\n  }\n}\n\nmain();\nglobalOne();\nglobalTwo();\n
\n

This prints “one” then “two”. In this example, it’s pretty clear that the two\na variables are different. But it’s not always so obvious. Consider:

\n
var globalOne;\nvar globalTwo;\n\nfun main() {\n  for (var a = 1; a <= 2; a = a + 1) {\n    fun closure() {\n      print a;\n    }\n    if (globalOne == nil) {\n      globalOne = closure;\n    } else {\n      globalTwo = closure;\n    }\n  }\n}\n\nmain();\nglobalOne();\nglobalTwo();\n
\n

The code is convoluted because Lox has no collection types. The important part\nis that the main() function does two iterations of a for loop. Each time\nthrough the loop, it creates a closure that captures the loop variable. It\nstores the first closure in globalOne and the second in globalTwo.

\n

There are definitely two different closures. Do they close over two different\nvariables? Is there only one a for the entire duration of the loop, or does\neach iteration get its own distinct a variable?

\n

The script here is strange and contrived, but this does show up in real code\nin languages that aren’t as minimal as clox. Here’s a JavaScript example:

\n
var closures = [];\nfor (var i = 1; i <= 2; i++) {\n  closures.push(function () { console.log(i); });\n}\n\nclosures[0]();\nclosures[1]();\n
\n

Does this print “1” then “2”, or does it print “3”\ntwice? You may be surprised to hear that it prints “3” twice. In this JavaScript\nprogram, there is only a single i variable whose lifetime includes all\niterations of the loop, including the final exit.

\n\n

If you’re familiar with JavaScript, you probably know that variables declared\nusing var are implicitly hoisted to the surrounding function or top-level\nscope. It’s as if you really wrote this:

\n
var closures = [];\nvar i;\nfor (i = 1; i <= 2; i++) {\n  closures.push(function () { console.log(i); });\n}\n\nclosures[0]();\nclosures[1]();\n
\n

At that point, it’s clearer that there is only a single i. Now consider if\nyou change the program to use the newer let keyword:

\n
var closures = [];\nfor (let i = 1; i <= 2; i++) {\n  closures.push(function () { console.log(i); });\n}\n\nclosures[0]();\nclosures[1]();\n
\n

Does this new program behave the same? Nope. In this case, it prints “1” then\n“2”. Each closure gets its own i. That’s sort of strange when you think about\nit. The increment clause is i++. That looks very much like it is assigning to\nand mutating an existing variable, not creating a new one.

\n

Let’s try some other languages. Here’s Python:

\n
closures = []\nfor i in range(1, 3):\n  closures.append(lambda: print(i))\n\nclosures[0]()\nclosures[1]()\n
\n

Python doesn’t really have block scope. Variables are implicitly declared and\nare automatically scoped to the surrounding function. Kind of like hoisting in\nJS, now that I think about it. So both closures capture the same variable.\nUnlike C, though, we don’t exit the loop by incrementing i past the last\nvalue, so this prints “2” twice.

\n

What about Ruby? Ruby has two typical ways to iterate numerically. Here’s the\nclassic imperative style:

\n
closures = []\nfor i in 1..2 do\n  closures << lambda { puts i }\nend\n\nclosures[0].call\nclosures[1].call\n
\n

This, like Python, prints “2” twice. But the more idiomatic Ruby style is using\na higher-order each() method on range objects:

\n
closures = []\n(1..2).each do |i|\n  closures << lambda { puts i }\nend\n\nclosures[0].call\nclosures[1].call\n
\n

If you’re not familiar with Ruby, the do |i| ... end part is basically a\nclosure that gets created and passed to the each() method. The |i| is the\nparameter signature for the closure. The each() method invokes that closure\ntwice, passing in 1 for i the first time and 2 the second time.

\n

In this case, the “loop variable” is really a function parameter. And, since\neach iteration of the loop is a separate invocation of the function, those are\ndefinitely separate variables for each call. So this prints “1” then “2”.

\n

If a language has a higher-level iterator-based looping structure like foreach\nin C#, Java’s “enhanced for”, for-of in JavaScript, for-in in Dart, etc.,\nthen I think it’s natural to the reader to have each iteration create a new\nvariable. The code looks like a new variable because the loop header looks\nlike a variable declaration. And there’s no increment expression that looks like\nit’s mutating that variable to advance to the next step.

\n

If you dig around StackOverflow and other places, you find evidence that this is\nwhat users expect, because they are very surprised when they don’t get it. In\nparticular, C# originally did not create a new loop variable for each\niteration of a foreach loop. This was such a frequent source of user confusion\nthat they took the very rare step of shipping a breaking change to the language.\nIn C# 5, each iteration creates a fresh variable.

\n

Old C-style for loops are harder. The increment clause really does look like\nmutation. That implies there is a single variable that’s getting updated each\nstep. But it’s almost never useful for each iteration to share a loop\nvariable. The only time you can even detect this is when closures capture it.\nAnd it’s rarely helpful to have a closure that references a variable whose value\nis whatever value caused you to exit the loop.

\n

The pragmatically useful answer is probably to do what JavaScript does with\nlet in for loops. Make it look like mutation but actually create a new\nvariable each time, because that’s what users want. It is kind of weird when you\nthink about it, though.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/compiling-expressions.html", "content": "\n\n\n\nCompiling Expressions · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
17
\n

Compiling Expressions

\n\n
\n

In the middle of the journey of our life I found myself within a dark woods\nwhere the straight way was lost.

\n

Dante Alighieri, Inferno

\n
\n

This chapter is exciting for not one, not two, but three reasons. First, it\nprovides the final segment of our VM’s execution pipeline. Once in place, we can\nplumb the user’s source code from scanning all the way through to executing it.

\"Lowering\n

Second, we get to write an actual, honest-to-God compiler. It parses source\ncode and outputs a low-level series of binary instructions. Sure, it’s bytecode and not some chip’s native instruction set, but\nit’s way closer to the metal than jlox was. We’re about to be real language\nhackers.

\n\n

Third and finally, I get to show you one of my\nabsolute favorite algorithms: Vaughan Pratt’s “top-down operator precedence\nparsing”. It’s the most elegant way I know to parse expressions. It gracefully\nhandles prefix operators, postfix, infix, mixfix, any kind of -fix you got.\nIt deals with precedence and associativity without breaking a sweat. I love it.

\n\n

As usual, before we get to the fun stuff, we’ve got some preliminaries to work\nthrough. You have to eat your vegetables before you get dessert. First, let’s\nditch that temporary scaffolding we wrote for testing the scanner and replace it\nwith something more useful.

\n
InterpretResult interpret(const char* source) {\n
vm.c
\nin interpret()
\nreplace 2 lines
\n
  Chunk chunk;\n  initChunk(&chunk);\n\n  if (!compile(source, &chunk)) {\n    freeChunk(&chunk);\n    return INTERPRET_COMPILE_ERROR;\n  }\n\n  vm.chunk = &chunk;\n  vm.ip = vm.chunk->code;\n\n  InterpretResult result = run();\n\n  freeChunk(&chunk);\n  return result;\n
}\n
\n
vm.c, in interpret(), replace 2 lines
\n\n

We create a new empty chunk and pass it over to the compiler. The compiler will\ntake the user’s program and fill up the chunk with bytecode. At least, that’s\nwhat it will do if the program doesn’t have any compile errors. If it does\nencounter an error, compile() returns false and we discard the unusable\nchunk.

\n

Otherwise, we send the completed chunk over to the VM to be executed. When the\nVM finishes, we free the chunk and we’re done. As you can see, the signature to\ncompile() is different now.

\n
#define clox_compiler_h\n\n
compiler.h
\nreplace 1 line
\n
#include "vm.h"\n\nbool compile(const char* source, Chunk* chunk);\n
\n\n#endif\n
\n
compiler.h, replace 1 line
\n\n

We pass in the chunk where the compiler will write the code, and then\ncompile() returns whether or not compilation succeeded. We make the same\nchange to the signature in the implementation.

\n
#include "scanner.h"\n\n
compiler.c
\nfunction compile()
\nreplace 1 line
\n
bool compile(const char* source, Chunk* chunk) {\n
  initScanner(source);\n
\n
compiler.c, function compile(), replace 1 line
\n\n

That call to initScanner() is the only line that survives this chapter. Rip\nout the temporary code we wrote to test the scanner and replace it with these\nthree lines:

\n
  initScanner(source);\n
compiler.c
\nin compile()
\nreplace 13 lines
\n
  advance();\n  expression();\n  consume(TOKEN_EOF, "Expect end of expression.");\n
}\n
\n
compiler.c, in compile(), replace 13 lines
\n\n

The call to advance() “primes the pump” on the scanner. We’ll see what it does\nsoon. Then we parse a single expression. We aren’t going to do statements yet,\nso that’s the only subset of the grammar we support. We’ll revisit this when we\nadd statements in a few chapters. After we compile the expression, we\nshould be at the end of the source code, so we check for the sentinel EOF token.

\n

We’re going to spend the rest of the chapter making this function work,\nespecially that little expression() call. Normally, we’d dive right into that\nfunction definition and work our way through the implementation from top to\nbottom.

\n

This chapter is different. Pratt’s parsing technique is\nremarkably simple once you have it all loaded in your head, but it’s a little\ntricky to break into bite-sized pieces. It’s recursive, of course, which is part\nof the problem. But it also relies on a big table of data. As we build up the\nalgorithm, that table grows additional columns.

\n\n

I don’t want to revisit 40-something lines of code each time we extend the\ntable. So we’re going to work our way into the core of the parser from the\noutside and cover all of the surrounding bits before we get to the juicy center.\nThis will require a little more patience and mental scratch space than most\nchapters, but it’s the best I could do.

\n

17 . 1Single-Pass Compilation

\n

A compiler has roughly two jobs. It parses the user’s source code to understand\nwhat it means. Then it takes that knowledge and outputs low-level instructions\nthat produce the same semantics. Many languages split those two roles into two\nseparate passes in the implementation. A parser\nproduces an AST—just like jlox does—and then a code generator traverses\nthe AST and outputs target code.

\n\n

In clox, we’re taking an old-school approach and merging these two passes into\none. Back in the day, language hackers did this because computers literally\ndidn’t have enough memory to store an entire source file’s AST. We’re doing it\nbecause it keeps our compiler simpler, which is a real asset when programming in\nC.

\n

Single-pass compilers like we’re going to build don’t work well for all\nlanguages. Since the compiler has only a peephole view into the user’s program\nwhile generating code, the language must be designed such that you don’t need\nmuch surrounding context to understand a piece of syntax. Fortunately, tiny,\ndynamically typed Lox is well-suited to that.

\n\n

What this means in practical terms is that our “compiler” C module has\nfunctionality you’ll recognize from jlox for parsing—consuming tokens,\nmatching expected token types, etc. And it also has functions for code gen—emitting bytecode and adding constants to the destination chunk. (And it means\nI’ll use “parsing” and “compiling” interchangeably throughout this and later\nchapters.)

\n

We’ll build the parsing and code generation halves first. Then we’ll stitch them\ntogether with the code in the middle that uses Pratt’s technique to parse Lox’s\nparticular grammar and output the right bytecode.

\n

17 . 2Parsing Tokens

\n

First up, the front half of the compiler. This function’s name should sound\nfamiliar.

\n
#include "scanner.h"\n
compiler.c
\n
\n\nstatic void advance() {\n  parser.previous = parser.current;\n\n  for (;;) {\n    parser.current = scanToken();\n    if (parser.current.type != TOKEN_ERROR) break;\n\n    errorAtCurrent(parser.current.start);\n  }\n}\n
\n
compiler.c
\n\n

Just like in jlox, it steps forward through the token stream. It asks the\nscanner for the next token and stores it for later use. Before doing that, it\ntakes the old current token and stashes that in a previous field. That will\ncome in handy later so that we can get at the lexeme after we match a token.

\n

The code to read the next token is wrapped in a loop. Remember, clox’s scanner\ndoesn’t report lexical errors. Instead, it creates special error tokens and\nleaves it up to the parser to report them. We do that here.

\n

We keep looping, reading tokens and reporting the errors, until we hit a\nnon-error one or reach the end. That way, the rest of the parser sees only valid\ntokens. The current and previous token are stored in this struct:

\n
#include "scanner.h"\n
compiler.c
\n
\n\ntypedef struct {\n  Token current;\n  Token previous;\n} Parser;\n\nParser parser;\n
\n\nstatic void advance() {\n
\n
compiler.c
\n\n

Like we did in other modules, we have a single global variable of this struct\ntype so we don’t need to pass the state around from function to function in the\ncompiler.

\n

17 . 2 . 1Handling syntax errors

\n

If the scanner hands us an error token, we need to actually tell the user. That\nhappens using this:

\n
compiler.c
\nadd after variable parser
\n
static void errorAtCurrent(const char* message) {\n  errorAt(&parser.current, message);\n}\n
\n
compiler.c, add after variable parser
\n\n

We pull the location out of the current token in order to tell the user where\nthe error occurred and forward it to errorAt(). More often, we’ll report an\nerror at the location of the token we just consumed, so we give the shorter name\nto this other function:

\n
compiler.c
\nadd after variable parser
\n
static void error(const char* message) {\n  errorAt(&parser.previous, message);\n}\n
\n
compiler.c, add after variable parser
\n\n

The actual work happens here:

\n
compiler.c
\nadd after variable parser
\n
static void errorAt(Token* token, const char* message) {\n  fprintf(stderr, "[line %d] Error", token->line);\n\n  if (token->type == TOKEN_EOF) {\n    fprintf(stderr, " at end");\n  } else if (token->type == TOKEN_ERROR) {\n    // Nothing.\n  } else {\n    fprintf(stderr, " at '%.*s'", token->length, token->start);\n  }\n\n  fprintf(stderr, ": %s\\n", message);\n  parser.hadError = true;\n}\n
\n
compiler.c, add after variable parser
\n\n

First, we print where the error occurred. We try to show the lexeme if it’s\nhuman-readable. Then we print the error message itself. After that, we set this\nhadError flag. That records whether any errors occurred during compilation.\nThis field also lives in the parser struct.

\n
  Token previous;\n
compiler.c
\nin struct Parser
\n
  bool hadError;\n
} Parser;\n
\n
compiler.c, in struct Parser
\n\n

Earlier I said that compile() should return false if an error occurred. Now\nwe can make it do that.

\n
  consume(TOKEN_EOF, "Expect end of expression.");\n
compiler.c
\nin compile()
\n
  return !parser.hadError;\n
}\n
\n
compiler.c, in compile()
\n\n

I’ve got another flag to introduce for error handling. We want to avoid error\ncascades. If the user has a mistake in their code and the parser gets confused\nabout where it is in the grammar, we don’t want it to spew out a whole pile of\nmeaningless knock-on errors after the first one.

\n

We fixed that in jlox using panic mode error recovery. In the Java interpreter,\nwe threw an exception to unwind out of all of the parser code to a point where\nwe could skip tokens and resynchronize. We don’t have exceptions in C. Instead, we’ll do a little smoke and\nmirrors. We add a flag to track whether we’re currently in panic mode.

\n\n
  bool hadError;\n
compiler.c
\nin struct Parser
\n
  bool panicMode;\n
} Parser;\n
\n
compiler.c, in struct Parser
\n\n

When an error occurs, we set it.

\n
static void errorAt(Token* token, const char* message) {\n
compiler.c
\nin errorAt()
\n
  parser.panicMode = true;\n
  fprintf(stderr, "[line %d] Error", token->line);\n
\n
compiler.c, in errorAt()
\n\n

After that, we go ahead and keep compiling as normal as if the error never\noccurred. The bytecode will never get executed, so it’s harmless to keep on\ntrucking. The trick is that while the panic mode flag is set, we simply suppress\nany other errors that get detected.

\n
static void errorAt(Token* token, const char* message) {\n
compiler.c
\nin errorAt()
\n
  if (parser.panicMode) return;\n
  parser.panicMode = true;\n
\n
compiler.c, in errorAt()
\n\n

There’s a good chance the parser will go off in the weeds, but the user won’t\nknow because the errors all get swallowed. Panic mode ends when the parser\nreaches a synchronization point. For Lox, we chose statement boundaries, so when\nwe later add those to our compiler, we’ll clear the flag there.

\n

These new fields need to be initialized.

\n
  initScanner(source);\n
compiler.c
\nin compile()
\n
\n\n  parser.hadError = false;\n  parser.panicMode = false;\n\n
  advance();\n
\n
compiler.c, in compile()
\n\n

And to display the errors, we need a standard header.

\n
#include <stdio.h>\n
compiler.c
\n
#include <stdlib.h>\n
\n\n#include "common.h"\n
\n
compiler.c
\n\n

There’s one last parsing function, another old friend from jlox.

\n
compiler.c
\nadd after advance()
\n
static void consume(TokenType type, const char* message) {\n  if (parser.current.type == type) {\n    advance();\n    return;\n  }\n\n  errorAtCurrent(message);\n}\n
\n
compiler.c, add after advance()
\n\n

It’s similar to advance() in that it reads the next token. But it also\nvalidates that the token has an expected type. If not, it reports an error. This\nfunction is the foundation of most syntax errors in the compiler.

\n

OK, that’s enough on the front end for now.

\n

17 . 3Emitting Bytecode

\n

After we parse and understand a piece of the user’s program, the next step is to\ntranslate that to a series of bytecode instructions. It starts with the easiest\npossible step: appending a single byte to the chunk.

\n
compiler.c
\nadd after consume()
\n
static void emitByte(uint8_t byte) {\n  writeChunk(currentChunk(), byte, parser.previous.line);\n}\n
\n
compiler.c, add after consume()
\n\n

It’s hard to believe great things will flow through such a simple function. It\nwrites the given byte, which may be an opcode or an operand to an instruction.\nIt sends in the previous token’s line information so that runtime errors are\nassociated with that line.

\n

The chunk that we’re writing gets passed into compile(), but it needs to make\nits way to emitByte(). To do that, we rely on this intermediary function:

\n
Parser parser;\n
compiler.c
\nadd after variable parser
\n
Chunk* compilingChunk;\n\nstatic Chunk* currentChunk() {\n  return compilingChunk;\n}\n\n
static void errorAt(Token* token, const char* message) {\n
\n
compiler.c, add after variable parser
\n\n

Right now, the chunk pointer is stored in a module-level variable like we store\nother global state. Later, when we start compiling user-defined functions, the\nnotion of “current chunk” gets more complicated. To avoid having to go back and\nchange a lot of code, I encapsulate that logic in the currentChunk() function.

\n

We initialize this new module variable before we write any bytecode:

\n
bool compile(const char* source, Chunk* chunk) {\n  initScanner(source);\n
compiler.c
\nin compile()
\n
  compilingChunk = chunk;\n
\n\n  parser.hadError = false;\n
\n
compiler.c, in compile()
\n\n

Then, at the very end, when we’re done compiling the chunk, we wrap things up.

\n
  consume(TOKEN_EOF, "Expect end of expression.");\n
compiler.c
\nin compile()
\n
  endCompiler();\n
  return !parser.hadError;\n
\n
compiler.c, in compile()
\n\n

That calls this:

\n
compiler.c
\nadd after emitByte()
\n
static void endCompiler() {\n  emitReturn();\n}\n
\n
compiler.c, add after emitByte()
\n\n

In this chapter, our VM deals only with expressions. When you run clox, it will\nparse, compile, and execute a single expression, then print the result. To print\nthat value, we are temporarily using the OP_RETURN instruction. So we have the\ncompiler add one of those to the end of the chunk.

\n
compiler.c
\nadd after emitByte()
\n
static void emitReturn() {\n  emitByte(OP_RETURN);\n}\n
\n
compiler.c, add after emitByte()
\n\n

While we’re here in the back end we may as well make our lives easier.

\n
compiler.c
\nadd after emitByte()
\n
static void emitBytes(uint8_t byte1, uint8_t byte2) {\n  emitByte(byte1);\n  emitByte(byte2);\n}\n
\n
compiler.c, add after emitByte()
\n\n

Over time, we’ll have enough cases where we need to write an opcode followed by\na one-byte operand that it’s worth defining this convenience function.

\n

17 . 4Parsing Prefix Expressions

\n

We’ve assembled our parsing and code generation utility functions. The missing\npiece is the code in the middle that connects those together.

\"Parsing\n

The only step in compile() that we have left to implement is this function:

\n
compiler.c
\nadd after endCompiler()
\n
static void expression() {\n  // What goes here?\n}\n
\n
compiler.c, add after endCompiler()
\n\n

We aren’t ready to implement every kind of expression in Lox yet. Heck, we don’t\neven have Booleans. For this chapter, we’re only going to worry about four:

\n
    \n
  • Number literals: 123
    • \n
  • Parentheses for grouping: (123)
    • \n
  • Unary negation: -123
    • \n
  • The Four Horsemen of the Arithmetic: +, -, *, /
    • \n
\n

As we work through the functions to compile each of those kinds of expressions,\nwe’ll also assemble the requirements for the table-driven parser that calls\nthem.

\n

17 . 4 . 1Parsers for tokens

\n

For now, let’s focus on the Lox expressions that are each only a single token.\nIn this chapter, that’s just number literals, but there will be more later. Here’s\nhow we can compile them:

\n

We map each token type to a different kind of expression. We define a function\nfor each expression that outputs the appropriate bytecode. Then we build an\narray of function pointers. The indexes in the array correspond to the\nTokenType enum values, and the function at each index is the code to compile\nan expression of that token type.

\n

To compile number literals, we store a pointer to the following function at the\nTOKEN_NUMBER index in the array.

\n
compiler.c
\nadd after endCompiler()
\n
static void number() {\n  double value = strtod(parser.previous.start, NULL);\n  emitConstant(value);\n}\n
\n
compiler.c, add after endCompiler()
\n\n

We assume the token for the number literal has already been consumed and is\nstored in previous. We take that lexeme and use the C standard library to\nconvert it to a double value. Then we generate the code to load that value using\nthis function:

\n
compiler.c
\nadd after emitReturn()
\n
static void emitConstant(Value value) {\n  emitBytes(OP_CONSTANT, makeConstant(value));\n}\n
\n
compiler.c, add after emitReturn()
\n\n

First, we add the value to the constant table, then we emit an OP_CONSTANT\ninstruction that pushes it onto the stack at runtime. To insert an entry in the\nconstant table, we rely on:

\n
compiler.c
\nadd after emitReturn()
\n
static uint8_t makeConstant(Value value) {\n  int constant = addConstant(currentChunk(), value);\n  if (constant > UINT8_MAX) {\n    error("Too many constants in one chunk.");\n    return 0;\n  }\n\n  return (uint8_t)constant;\n}\n
\n
compiler.c, add after emitReturn()
\n\n

Most of the work happens in addConstant(), which we defined back in an\nearlier chapter. That adds the given value to the end of the chunk’s\nconstant table and returns its index. The new function’s job is mostly to make\nsure we don’t have too many constants. Since the OP_CONSTANT instruction uses\na single byte for the index operand, we can store and load only up to 256 constants in a chunk.

\n\n

That’s basically all it takes. Provided there is some suitable code that\nconsumes a TOKEN_NUMBER token, looks up number() in the function pointer\narray, and then calls it, we can now compile number literals to bytecode.

\n

17 . 4 . 2Parentheses for grouping

\n

Our as-yet-imaginary array of parsing function pointers would be great if every\nexpression was only a single token long. Alas, most are longer. However, many\nexpressions start with a particular token. We call these prefix expressions.\nFor example, when we’re parsing an expression and the current token is (, we\nknow we must be looking at a parenthesized grouping expression.

\n

It turns out our function pointer array handles those too. The parsing function\nfor an expression type can consume any additional tokens that it wants to, just\nlike in a regular recursive descent parser. Here’s how parentheses work:

\n
compiler.c
\nadd after endCompiler()
\n
static void grouping() {\n  expression();\n  consume(TOKEN_RIGHT_PAREN, "Expect ')' after expression.");\n}\n
\n
compiler.c, add after endCompiler()
\n\n

Again, we assume the initial ( has already been consumed. We recursively call back into expression() to compile the\nexpression between the parentheses, then parse the closing ) at the end.

\n\n

As far as the back end is concerned, there’s literally nothing to a grouping\nexpression. Its sole function is syntactic—it lets you insert a\nlower-precedence expression where a higher precedence is expected. Thus, it has\nno runtime semantics on its own and therefore doesn’t emit any bytecode. The\ninner call to expression() takes care of generating bytecode for the\nexpression inside the parentheses.

\n

17 . 4 . 3Unary negation

\n

Unary minus is also a prefix expression, so it works with our model too.

\n
compiler.c
\nadd after number()
\n
static void unary() {\n  TokenType operatorType = parser.previous.type;\n\n  // Compile the operand.\n  expression();\n\n  // Emit the operator instruction.\n  switch (operatorType) {\n    case TOKEN_MINUS: emitByte(OP_NEGATE); break;\n    default: return; // Unreachable.\n  }\n}\n
\n
compiler.c, add after number()
\n\n

The leading - token has been consumed and is sitting in parser.previous. We\ngrab the token type from that to note which unary operator we’re dealing with.\nIt’s unnecessary right now, but this will make more sense when we use this same\nfunction to compile the ! operator in the next chapter.

\n

As in grouping(), we recursively call expression() to compile the operand.\nAfter that, we emit the bytecode to perform the negation. It might seem a little\nweird to write the negate instruction after its operand’s bytecode since the\n- appears on the left, but think about it in terms of order of execution:

\n
    \n
  1. \n

    We evaluate the operand first which leaves its value on the stack.

    \n
    2. \n
  2. \n

    Then we pop that value, negate it, and push the result.

    \n
    3. \n
\n

So the OP_NEGATE instruction should be emitted last.\nThis is part of the compiler’s job—parsing the program in the order it\nappears in the source code and rearranging it into the order that execution\nhappens.

\n\n

There is one problem with this code, though. The expression() function it\ncalls will parse any expression for the operand, regardless of precedence. Once\nwe add binary operators and other syntax, that will do the wrong thing.\nConsider:

\n
-a.b + c;\n
\n

Here, the operand to - should be just the a.b expression, not the entire\na.b + c. But if unary() calls expression(), the latter will happily chew\nthrough all of the remaining code including the +. It will erroneously treat\nthe - as lower precedence than the +.

\n

When parsing the operand to unary -, we need to compile only expressions at a\ncertain precedence level or higher. In jlox’s recursive descent parser we\naccomplished that by calling into the parsing method for the lowest-precedence\nexpression we wanted to allow (in this case, call()). Each method for parsing\na specific expression also parsed any expressions of higher precedence too, so\nthat included the rest of the precedence table.

\n

The parsing functions like number() and unary() here in clox are different.\nEach only parses exactly one type of expression. They don’t cascade to include\nhigher-precedence expression types too. We need a different solution, and it\nlooks like this:

\n
compiler.c
\nadd after unary()
\n
static void parsePrecedence(Precedence precedence) {\n  // What goes here?\n}\n
\n
compiler.c, add after unary()
\n\n

This function—once we implement it—starts at the current token and parses\nany expression at the given precedence level or higher. We have some other setup\nto get through before we can write the body of this function, but you can\nprobably guess that it will use that table of parsing function pointers I’ve\nbeen talking about. For now, don’t worry too much about how it works. In order\nto take the “precedence” as a parameter, we define it numerically.

\n
} Parser;\n
compiler.c
\nadd after struct Parser
\n
\n\ntypedef enum {\n  PREC_NONE,\n  PREC_ASSIGNMENT,  // =\n  PREC_OR,          // or\n  PREC_AND,         // and\n  PREC_EQUALITY,    // == !=\n  PREC_COMPARISON,  // < > <= >=\n  PREC_TERM,        // + -\n  PREC_FACTOR,      // * /\n  PREC_UNARY,       // ! -\n  PREC_CALL,        // . ()\n  PREC_PRIMARY\n} Precedence;\n
\n\nParser parser;\n
\n
compiler.c, add after struct Parser
\n\n

These are all of Lox’s precedence levels in order from lowest to highest. Since\nC implicitly gives successively larger numbers for enums, this means that\nPREC_CALL is numerically larger than PREC_UNARY. For example, say the\ncompiler is sitting on a chunk of code like:

\n
-a.b + c\n
\n

If we call parsePrecedence(PREC_ASSIGNMENT), then it will parse the entire\nexpression because + has higher precedence than assignment. If instead we\ncall parsePrecedence(PREC_UNARY), it will compile the -a.b and stop there.\nIt doesn’t keep going through the + because the addition has lower precedence\nthan unary operators.

\n

With this function in hand, it’s a snap to fill in the missing body for\nexpression().

\n
static void expression() {\n
compiler.c
\nin expression()
\nreplace 1 line
\n
  parsePrecedence(PREC_ASSIGNMENT);\n
}\n
\n
compiler.c, in expression(), replace 1 line
\n\n

We simply parse the lowest precedence level, which subsumes all of the\nhigher-precedence expressions too. Now, to compile the operand for a unary\nexpression, we call this new function and limit it to the appropriate level:

\n
  // Compile the operand.\n
compiler.c
\nin unary()
\nreplace 1 line
\n
  parsePrecedence(PREC_UNARY);\n
\n\n  // Emit the operator instruction.\n
\n
compiler.c, in unary(), replace 1 line
\n\n

We use the unary operator’s own PREC_UNARY precedence to permit nested unary expressions like !!doubleNegative. Since\nunary operators have pretty high precedence, that correctly excludes things like\nbinary operators. Speaking of which . . . 

\n\n

17 . 5Parsing Infix Expressions

\n

Binary operators are different from the previous expressions because they are\ninfix. With the other expressions, we know what we are parsing from the very\nfirst token. With infix expressions, we don’t know we’re in the middle of a\nbinary operator until after we’ve parsed its left operand and then stumbled\nonto the operator token in the middle.

\n

Here’s an example:

\n
1 + 2\n
\n

Let’s walk through trying to compile it with what we know so far:

\n
    \n
  1. \n

    We call expression(). That in turn calls\nparsePrecedence(PREC_ASSIGNMENT).

    \n
    2. \n
  2. \n

    That function (once we implement it) sees the leading number token and\nrecognizes it is parsing a number literal. It hands off control to\nnumber().

    \n
    3. \n
  3. \n

    number() creates a constant, emits an OP_CONSTANT, and returns back to\nparsePrecedence().

    \n
    4. \n
\n

Now what? The call to parsePrecedence() should consume the entire addition\nexpression, so it needs to keep going somehow. Fortunately, the parser is right\nwhere we need it to be. Now that we’ve compiled the leading number expression,\nthe next token is +. That’s the exact token that parsePrecedence() needs to\ndetect that we’re in the middle of an infix expression and to realize that the\nexpression we already compiled is actually an operand to that.

\n

So this hypothetical array of function pointers doesn’t just list functions to\nparse expressions that start with a given token. Instead, it’s a table of\nfunction pointers. One column associates prefix parser functions with token\ntypes. The second column associates infix parser functions with token types.

\n

The function we will use as the infix parser for TOKEN_PLUS, TOKEN_MINUS,\nTOKEN_STAR, and TOKEN_SLASH is this:

\n
compiler.c
\nadd after endCompiler()
\n
static void binary() {\n  TokenType operatorType = parser.previous.type;\n  ParseRule* rule = getRule(operatorType);\n  parsePrecedence((Precedence)(rule->precedence + 1));\n\n  switch (operatorType) {\n    case TOKEN_PLUS:          emitByte(OP_ADD); break;\n    case TOKEN_MINUS:         emitByte(OP_SUBTRACT); break;\n    case TOKEN_STAR:          emitByte(OP_MULTIPLY); break;\n    case TOKEN_SLASH:         emitByte(OP_DIVIDE); break;\n    default: return; // Unreachable.\n  }\n}\n
\n
compiler.c, add after endCompiler()
\n\n

When a prefix parser function is called, the leading token has already been\nconsumed. An infix parser function is even more in medias res—the entire\nleft-hand operand expression has already been compiled and the subsequent infix\noperator consumed.

\n

The fact that the left operand gets compiled first works out fine. It means at\nruntime, that code gets executed first. When it runs, the value it produces will\nend up on the stack. That’s right where the infix operator is going to need it.

\n

Then we come here to binary() to handle the rest of the arithmetic operators.\nThis function compiles the right operand, much like how unary() compiles its\nown trailing operand. Finally, it emits the bytecode instruction that performs\nthe binary operation.

\n

When run, the VM will execute the left and right operand code, in that order,\nleaving their values on the stack. Then it executes the instruction for the\noperator. That pops the two values, computes the operation, and pushes the\nresult.

\n

The code that probably caught your eye here is that getRule() line. When we\nparse the right-hand operand, we again need to worry about precedence. Take an\nexpression like:

\n
2 * 3 + 4\n
\n

When we parse the right operand of the * expression, we need to just capture\n3, and not 3 + 4, because + is lower precedence than *. We could define\na separate function for each binary operator. Each would call\nparsePrecedence() and pass in the correct precedence level for its operand.

\n

But that’s kind of tedious. Each binary operator’s right-hand operand precedence\nis one level higher than its own. We can look that up\ndynamically with this getRule() thing we’ll get to soon. Using that, we call\nparsePrecedence() with one level higher than this operator’s level.

\n\n

This way, we can use a single binary() function for all binary operators even\nthough they have different precedences.

\n

17 . 6A Pratt Parser

\n

We now have all of the pieces and parts of the compiler laid out. We have a\nfunction for each grammar production: number(), grouping(), unary(), and\nbinary(). We still need to implement parsePrecedence(), and getRule(). We\nalso know we need a table that, given a token type, lets us find

\n
    \n
  • \n

    the function to compile a prefix expression starting with a token of that\ntype,

    \n
    • \n
  • \n

    the function to compile an infix expression whose left operand is followed\nby a token of that type, and

    \n
    • \n
  • \n

    the precedence of an infix expression that uses\nthat token as an operator.

    \n
    • \n
\n\n

We wrap these three properties in a little struct which represents a single row\nin the parser table.

\n
} Precedence;\n
compiler.c
\nadd after enum Precedence
\n
\n\ntypedef struct {\n  ParseFn prefix;\n  ParseFn infix;\n  Precedence precedence;\n} ParseRule;\n
\n\nParser parser;\n
\n
compiler.c, add after enum Precedence
\n\n

That ParseFn type is a simple typedef for a function\ntype that takes no arguments and returns nothing.

\n\n
} Precedence;\n
compiler.c
\nadd after enum Precedence
\n
\n\ntypedef void (*ParseFn)();\n
\n\ntypedef struct {\n
\n
compiler.c, add after enum Precedence
\n\n

The table that drives our whole parser is an array of ParseRules. We’ve been\ntalking about it forever, and finally you get to see it.

\n
compiler.c
\nadd after unary()
\n
ParseRule rules[] = {\n  [TOKEN_LEFT_PAREN]    = {grouping, NULL,   PREC_NONE},\n  [TOKEN_RIGHT_PAREN]   = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_LEFT_BRACE]    = {NULL,     NULL,   PREC_NONE}, \n  [TOKEN_RIGHT_BRACE]   = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_COMMA]         = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_DOT]           = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_MINUS]         = {unary,    binary, PREC_TERM},\n  [TOKEN_PLUS]          = {NULL,     binary, PREC_TERM},\n  [TOKEN_SEMICOLON]     = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_SLASH]         = {NULL,     binary, PREC_FACTOR},\n  [TOKEN_STAR]          = {NULL,     binary, PREC_FACTOR},\n  [TOKEN_BANG]          = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_BANG_EQUAL]    = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_EQUAL]         = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_EQUAL_EQUAL]   = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_GREATER]       = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_GREATER_EQUAL] = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_LESS]          = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_LESS_EQUAL]    = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_IDENTIFIER]    = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_STRING]        = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_NUMBER]        = {number,   NULL,   PREC_NONE},\n  [TOKEN_AND]           = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_CLASS]         = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_ELSE]          = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_FALSE]         = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_FOR]           = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_FUN]           = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_IF]            = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_NIL]           = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_OR]            = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_PRINT]         = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_RETURN]        = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_SUPER]         = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_THIS]          = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_TRUE]          = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_VAR]           = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_WHILE]         = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_ERROR]         = {NULL,     NULL,   PREC_NONE},\n  [TOKEN_EOF]           = {NULL,     NULL,   PREC_NONE},\n};\n
\n
compiler.c, add after unary()
\n\n\n

You can see how grouping and unary are slotted into the prefix parser column\nfor their respective token types. In the next column, binary is wired up to\nthe four arithmetic infix operators. Those infix operators also have their\nprecedences set in the last column.

\n

Aside from those, the rest of the table is full of NULL and PREC_NONE. Most\nof those empty cells are because there is no expression associated with those\ntokens. You can’t start an expression with, say, else, and } would make for\na pretty confusing infix operator.

\n

But, also, we haven’t filled in the entire grammar yet. In later chapters, as we\nadd new expression types, some of these slots will get functions in them. One of\nthe things I like about this approach to parsing is that it makes it very easy\nto see which tokens are in use by the grammar and which are available.

\n

Now that we have the table, we are finally ready to write the code that uses it.\nThis is where our Pratt parser comes to life. The easiest function to define is\ngetRule().

\n
compiler.c
\nadd after parsePrecedence()
\n
static ParseRule* getRule(TokenType type) {\n  return &rules[type];\n}\n
\n
compiler.c, add after parsePrecedence()
\n\n

It simply returns the rule at the given index. It’s called by binary() to look\nup the precedence of the current operator. This function exists solely to handle\na declaration cycle in the C code. binary() is defined before the rules\ntable so that the table can store a pointer to it. That means the body of\nbinary() cannot access the table directly.

\n

Instead, we wrap the lookup in a function. That lets us forward declare\ngetRule() before the definition of binary(), and then define getRule() after the table. We’ll need a\ncouple of other forward declarations to handle the fact that our grammar is\nrecursive, so let’s get them all out of the way.

\n\n
  emitReturn();\n}\n
compiler.c
\nadd after endCompiler()
\n
\n\nstatic void expression();\nstatic ParseRule* getRule(TokenType type);\nstatic void parsePrecedence(Precedence precedence);\n\n
static void binary() {\n
\n
compiler.c, add after endCompiler()
\n\n

If you’re following along and implementing clox yourself, pay close attention to\nthe little annotations that tell you where to put these code snippets. Don’t\nworry, though, if you get it wrong, the C compiler will be happy to tell you.

\n

17 . 6 . 1Parsing with precedence

\n

Now we’re getting to the fun stuff. The maestro that orchestrates all of the\nparsing functions we’ve defined is parsePrecedence(). Let’s start with parsing\nprefix expressions.

\n
static void parsePrecedence(Precedence precedence) {\n
compiler.c
\nin parsePrecedence()
\nreplace 1 line
\n
  advance();\n  ParseFn prefixRule = getRule(parser.previous.type)->prefix;\n  if (prefixRule == NULL) {\n    error("Expect expression.");\n    return;\n  }\n\n  prefixRule();\n
}\n
\n
compiler.c, in parsePrecedence(), replace 1 line
\n\n

We read the next token and look up the corresponding ParseRule. If there is no\nprefix parser, then the token must be a syntax error. We report that and return\nto the caller.

\n

Otherwise, we call that prefix parse function and let it do its thing. That\nprefix parser compiles the rest of the prefix expression, consuming any other\ntokens it needs, and returns back here. Infix expressions are where it gets\ninteresting since precedence comes into play. The implementation is remarkably\nsimple.

\n
  prefixRule();\n
compiler.c
\nin parsePrecedence()
\n
\n\n  while (precedence <= getRule(parser.current.type)->precedence) {\n    advance();\n    ParseFn infixRule = getRule(parser.previous.type)->infix;\n    infixRule();\n  }\n
}\n
\n
compiler.c, in parsePrecedence()
\n\n

That’s the whole thing. Really. Here’s how the entire function works: At the\nbeginning of parsePrecedence(), we look up a prefix parser for the current\ntoken. The first token is always going to belong to some kind of prefix\nexpression, by definition. It may turn out to be nested as an operand inside one\nor more infix expressions, but as you read the code from left to right, the\nfirst token you hit always belongs to a prefix expression.

\n

After parsing that, which may consume more tokens, the prefix expression is\ndone. Now we look for an infix parser for the next token. If we find one, it\nmeans the prefix expression we already compiled might be an operand for it. But\nonly if the call to parsePrecedence() has a precedence that is low enough to\npermit that infix operator.

\n

If the next token is too low precedence, or isn’t an infix operator at all,\nwe’re done. We’ve parsed as much expression as we can. Otherwise, we consume the\noperator and hand off control to the infix parser we found. It consumes whatever\nother tokens it needs (usually the right operand) and returns back to\nparsePrecedence(). Then we loop back around and see if the next token is\nalso a valid infix operator that can take the entire preceding expression as its\noperand. We keep looping like that, crunching through infix operators and their\noperands until we hit a token that isn’t an infix operator or is too low\nprecedence and stop.

\n

That’s a lot of prose, but if you really want to mind meld with Vaughan Pratt\nand fully understand the algorithm, step through the parser in your debugger as\nit works through some expressions. Maybe a picture will help. There’s only a\nhandful of functions, but they are marvelously intertwined:

\n

\n

\"The

\n\n

Later, we’ll need to tweak the code in this chapter to handle assignment. But,\notherwise, what we wrote covers all of our expression compiling needs for the\nrest of the book. We’ll plug additional parsing functions into the table when we\nadd new kinds of expressions, but parsePrecedence() is complete.

\n

17 . 7Dumping Chunks

\n

While we’re here in the core of our compiler, we should put in some\ninstrumentation. To help debug the generated bytecode, we’ll add support for\ndumping the chunk once the compiler finishes. We had some temporary logging\nearlier when we hand-authored the chunk. Now we’ll put in some real code so that\nwe can enable it whenever we want.

\n

Since this isn’t for end users, we hide it behind a flag.

\n
#include <stdint.h>\n\n
common.h
\n
#define DEBUG_PRINT_CODE\n
#define DEBUG_TRACE_EXECUTION\n
\n
common.h
\n\n

When that flag is defined, we use our existing “debug” module to print out the\nchunk’s bytecode.

\n
  emitReturn();\n
compiler.c
\nin endCompiler()
\n
#ifdef DEBUG_PRINT_CODE\n  if (!parser.hadError) {\n    disassembleChunk(currentChunk(), "code");\n  }\n#endif\n
}\n
\n
compiler.c, in endCompiler()
\n\n

We do this only if the code was free of errors. After a syntax error, the\ncompiler keeps on going but it’s in kind of a weird state and might produce\nbroken code. That’s harmless because it won’t get executed, but we’ll just\nconfuse ourselves if we try to read it.

\n

Finally, to access disassembleChunk(), we need to include its header.

\n
#include "scanner.h"\n
compiler.c
\n
\n\n#ifdef DEBUG_PRINT_CODE\n#include "debug.h"\n#endif\n
\n\ntypedef struct {\n
\n
compiler.c
\n\n

We made it! This was the last major section to install in our VM’s compilation\nand execution pipeline. Our interpreter doesn’t look like much, but inside it\nis scanning, parsing, compiling to bytecode, and executing.

\n

Fire up the VM and type in an expression. If we did everything right, it should\ncalculate and print the result. We now have a very over-engineered arithmetic\ncalculator. We have a lot of language features to add in the coming chapters,\nbut the foundation is in place.

\n
\n

Challenges

\n
    \n
  1. \n

    To really understand the parser, you need to see how execution threads\nthrough the interesting parsing functions—parsePrecedence() and the\nparser functions stored in the table. Take this (strange) expression:

    \n
    (-1 + 2) * 3 - -4\n
    \n

    Write a trace of how those functions are called. Show the order they are\ncalled, which calls which, and the arguments passed to them.

    \n
    2. \n
  2. \n

    The ParseRule row for TOKEN_MINUS has both prefix and infix function\npointers. That’s because - is both a prefix operator (unary negation) and\nan infix one (subtraction).

    \n

    In the full Lox language, what other tokens can be used in both prefix and\ninfix positions? What about in C or in another language of your choice?

    \n
    3. \n
  3. \n

    You might be wondering about complex “mixfix” expressions that have more\nthan two operands separated by tokens. C’s conditional or “ternary”\noperator, ?:, is a widely known one.

    \n

    Add support for that operator to the compiler. You don’t have to generate\nany bytecode, just show how you would hook it up to the parser and handle\nthe operands.

    \n
    4. \n
\n
\n
\n

Design Note: It’s Just Parsing

\n

I’m going to make a claim here that will be unpopular with some compiler and\nlanguage people. It’s OK if you don’t agree. Personally, I learn more from\nstrongly stated opinions that I disagree with than I do from several pages of\nqualifiers and equivocation. My claim is that parsing doesn’t matter.

\n

Over the years, many programming language people, especially in academia, have\ngotten really into parsers and taken them very seriously. Initially, it was\nthe compiler folks who got into compiler-compilers,\nLALR, and other stuff like that. The first half of the dragon book is a long\nlove letter to the wonders of parser generators.

\n\n

Later, the functional programming folks got into parser combinators, packrat\nparsers, and other sorts of things. Because, obviously, if you give a functional\nprogrammer a problem, the first thing they’ll do is whip out a pocketful of\nhigher-order functions.

\n

Over in math and algorithm analysis land, there is a long legacy of research\ninto proving time and memory usage for various parsing techniques, transforming\nparsing problems into other problems and back, and assigning complexity classes\nto different grammars.

\n

At one level, this stuff is important. If you’re implementing a language, you\nwant some assurance that your parser won’t go exponential and take 7,000 years\nto parse a weird edge case in the grammar. Parser theory gives you that bound.\nAs an intellectual exercise, learning about parsing techniques is also fun and\nrewarding.

\n

But if your goal is just to implement a language and get it in front of users,\nalmost all of that stuff doesn’t matter. It’s really easy to get worked up by\nthe enthusiasm of the people who are into it and think that your front end\nneeds some whiz-bang generated combinator-parser-factory thing. I’ve seen\npeople burn tons of time writing and rewriting their parser using whatever\ntoday’s hot library or technique is.

\n

That’s time that doesn’t add any value to your user’s life. If you’re just\ntrying to get your parser done, pick one of the bog-standard techniques, use it,\nand move on. Recursive descent, Pratt parsing, and the popular parser generators\nlike ANTLR or Bison are all fine.

\n

Take the extra time you saved not rewriting your parsing code and spend it\nimproving the compile error messages your compiler shows users. Good error\nhandling and reporting is more valuable to users than almost anything else you\ncan put time into in the front end.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/contents.html", "content": "\n\n\n\nTable of Contents · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n

Table of Contents

\n\n
\n
\n
\n

❧Frontmatter

\n \n\n

I.Welcome

\n

II.A Tree-Walk Interpreter

\n
\n
\n

III.A Bytecode Virtual Machine

\n \n

❧Backmatter

\n \n
\n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/control-flow.html", "content": "\n\n\n\nControl Flow · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
9
\n

Control Flow

\n\n
\n

Logic, like whiskey, loses its beneficial effect when taken in too large\nquantities.

\n

Edward John Moreton Drax Plunkett, Lord Dunsany

\n
\n

Compared to last chapter’s grueling marathon, today is a\nlighthearted frolic through a daisy meadow. But while the work is easy, the\nreward is surprisingly large.

\n

Right now, our interpreter is little more than a calculator. A Lox program can\nonly do a fixed amount of work before completing. To make it run twice as long\nyou have to make the source code twice as lengthy. We’re about to fix that. In\nthis chapter, our interpreter takes a big step towards the programming\nlanguage major leagues: Turing-completeness.

\n

9 . 1Turing Machines (Briefly)

\n

In the early part of last century, mathematicians stumbled into a series of\nconfusing paradoxes that led them to doubt the\nstability of the foundation they had built their work upon. To address that\ncrisis, they went back to square one. Starting from a handful of axioms,\nlogic, and set theory, they hoped to rebuild mathematics on top of an\nimpervious foundation.

\n\n

They wanted to rigorously answer questions like, “Can all true statements be\nproven?”, “Can we compute all functions that we can define?”, or even the\nmore general question, “What do we mean when we claim a function is\n‘computable’?”

\n

They presumed the answer to the first two questions would be “yes”. All that\nremained was to prove it. It turns out that the answer to both is “no”, and\nastonishingly, the two questions are deeply intertwined. This is a fascinating\ncorner of mathematics that touches fundamental questions about what brains are\nable to do and how the universe works. I can’t do it justice here.

\n

What I do want to note is that in the process of proving that the answer to the\nfirst two questions is “no”, Alan Turing and Alonzo Church devised a precise\nanswer to the last question—a definition of what kinds of functions are computable. They each crafted a tiny system with a\nminimum set of machinery that is still powerful enough to compute any of a\n(very) large class of functions.

\n\n

These are now considered the “computable functions”. Turing’s system is called a\nTuring machine. Church’s is the lambda\ncalculus. Both are still widely used as the basis for models of computation\nand, in fact, many modern functional programming languages use the lambda\ncalculus at their core.

\n\"A\n

Turing machines have better name recognition—there’s no Hollywood film about\nAlonzo Church yet—but the two formalisms are equivalent in power.\nIn fact, any programming language with some minimal level of expressiveness is\npowerful enough to compute any computable function.

\n

You can prove that by writing a simulator for a Turing machine in your language.\nSince Turing proved his machine can compute any computable function, by\nextension, that means your language can too. All you need to do is translate the\nfunction into a Turing machine, and then run that on your simulator.

\n

If your language is expressive enough to do that, it’s considered\nTuring-complete. Turing machines are pretty dang simple, so it doesn’t take\nmuch power to do this. You basically need arithmetic, a little control flow,\nand the ability to allocate and use (theoretically) arbitrary amounts of memory.\nWe’ve got the first. By the end of this chapter, we’ll have the second.

\n\n

9 . 2Conditional Execution

\n

Enough history, let’s jazz up our language. We can divide control flow roughly\ninto two kinds:

\n
    \n
  • \n

    Conditional or branching control flow is used to not execute\nsome piece of code. Imperatively, you can think of it as jumping ahead\nover a region of code.

    \n
    • \n
  • \n

    Looping control flow executes a chunk of code more than once. It jumps\nback so that you can do something again. Since you don’t usually want\ninfinite loops, it typically has some conditional logic to know when to\nstop looping as well.

    \n
    • \n
\n

Branching is simpler, so we’ll start there. C-derived languages have two main\nconditional execution features, the if statement and the perspicaciously named\n“conditional” operator (?:). An if statement\nlets you conditionally execute statements and the conditional operator lets you\nconditionally execute expressions.

\n\n

For simplicity’s sake, Lox doesn’t have a conditional operator, so let’s get our\nif statement on. Our statement grammar gets a new production.

\n

\n
statement      → exprStmt\n               | ifStmt\n               | printStmt\n               | block ;\n\nifStmt         → "if" "(" expression ")" statement\n               ( "else" statement )? ;\n
\n\n

An if statement has an expression for the condition, then a statement to execute\nif the condition is truthy. Optionally, it may also have an else keyword and a\nstatement to execute if the condition is falsey. The syntax\ntree node has fields for each of those three pieces.

\n
      "Expression : Expr expression",\n
tool/GenerateAst.java
\nin main()
\n
      "If         : Expr condition, Stmt thenBranch," +\n                  " Stmt elseBranch",\n
      "Print      : Expr expression",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

Like other statements, the parser recognizes an if statement by the leading\nif keyword.

\n
  private Stmt statement() {\n
lox/Parser.java
\nin statement()
\n
    if (match(IF)) return ifStatement();\n
    if (match(PRINT)) return printStatement();\n
\n
lox/Parser.java, in statement()
\n\n

When it finds one, it calls this new method to parse the rest:

\n
lox/Parser.java
\nadd after statement()
\n
  private Stmt ifStatement() {\n    consume(LEFT_PAREN, "Expect '(' after 'if'.");\n    Expr condition = expression();\n    consume(RIGHT_PAREN, "Expect ')' after if condition."); \n\n    Stmt thenBranch = statement();\n    Stmt elseBranch = null;\n    if (match(ELSE)) {\n      elseBranch = statement();\n    }\n\n    return new Stmt.If(condition, thenBranch, elseBranch);\n  }\n
\n
lox/Parser.java, add after statement()
\n\n\n

As usual, the parsing code hews closely to the grammar. It detects an else\nclause by looking for the preceding else keyword. If there isn’t one, the\nelseBranch field in the syntax tree is null.

\n

That seemingly innocuous optional else has, in fact, opened up an ambiguity in\nour grammar. Consider:

\n
if (first) if (second) whenTrue(); else whenFalse();\n
\n

Here’s the riddle: Which if statement does that else clause belong to? This\nisn’t just a theoretical question about how we notate our grammar. It actually\naffects how the code executes:

\n
    \n
  • \n

    If we attach the else to the first if statement, then whenFalse() is\ncalled if first is falsey, regardless of what value second has.

    \n
    • \n
  • \n

    If we attach it to the second if statement, then whenFalse() is only\ncalled if first is truthy and second is falsey.

    \n
    • \n
\n

Since else clauses are optional, and there is no explicit delimiter marking the\nend of the if statement, the grammar is ambiguous when you nest ifs in this\nway. This classic pitfall of syntax is called the dangling else problem.

\n

\"Two\n\n

It is possible to define a context-free grammar that avoids the ambiguity\ndirectly, but it requires splitting most of the statement rules into pairs, one\nthat allows an if with an else and one that doesn’t. It’s annoying.

\n

Instead, most languages and parsers avoid the problem in an ad hoc way. No\nmatter what hack they use to get themselves out of the trouble, they always\nchoose the same interpretation—the else is bound to the nearest if that\nprecedes it.

\n

Our parser conveniently does that already. Since ifStatement() eagerly looks\nfor an else before returning, the innermost call to a nested series will claim\nthe else clause for itself before returning to the outer if statements.

\n

Syntax in hand, we are ready to interpret.

\n
lox/Interpreter.java
\nadd after visitExpressionStmt()
\n
  @Override\n  public Void visitIfStmt(Stmt.If stmt) {\n    if (isTruthy(evaluate(stmt.condition))) {\n      execute(stmt.thenBranch);\n    } else if (stmt.elseBranch != null) {\n      execute(stmt.elseBranch);\n    }\n    return null;\n  }\n
\n
lox/Interpreter.java, add after visitExpressionStmt()
\n\n

The interpreter implementation is a thin wrapper around the self-same Java code.\nIt evaluates the condition. If truthy, it executes the then branch. Otherwise,\nif there is an else branch, it executes that.

\n

If you compare this code to how the interpreter handles other syntax we’ve\nimplemented, the part that makes control flow special is that Java if\nstatement. Most other syntax trees always evaluate their subtrees. Here, we may\nnot evaluate the then or else statement. If either of those has a side effect,\nthe choice not to evaluate it becomes user visible.

\n

9 . 3Logical Operators

\n

Since we don’t have the conditional operator, you might think we’re done with\nbranching, but no. Even without the ternary operator, there are two other\noperators that are technically control flow constructs—the logical operators\nand and or.

\n

These aren’t like other binary operators because they short-circuit. If,\nafter evaluating the left operand, we know what the result of the logical\nexpression must be, we don’t evaluate the right operand. For example:

\n
false and sideEffect();\n
\n

For an and expression to evaluate to something truthy, both operands must be\ntruthy. We can see as soon as we evaluate the left false operand that that\nisn’t going to be the case, so there’s no need to evaluate sideEffect() and it\ngets skipped.

\n

This is why we didn’t implement the logical operators with the other binary\noperators. Now we’re ready. The two new operators are low in the precedence\ntable. Similar to || and && in C, they each have their own precedence with or lower than and. We slot them\nright between assignment and equality.

\n\n
expression     → assignment ;\nassignment     → IDENTIFIER "=" assignment\n               | logic_or ;\nlogic_or       → logic_and ( "or" logic_and )* ;\nlogic_and      → equality ( "and" equality )* ;\n
\n

Instead of falling back to equality, assignment now cascades to logic_or.\nThe two new rules, logic_or and logic_and, are similar to other binary operators. Then logic_and calls\nout to equality for its operands, and we chain back to the rest of the\nexpression rules.

\n\n

We could reuse the existing Expr.Binary class for these two new expressions\nsince they have the same fields. But then visitBinaryExpr() would have to\ncheck to see if the operator is one of the logical operators and use a different\ncode path to handle the short circuiting. I think it’s cleaner to define a new class for these operators so that they get their\nown visit method.

\n
      "Literal  : Object value",\n
tool/GenerateAst.java
\nin main()
\n
      "Logical  : Expr left, Token operator, Expr right",\n
      "Unary    : Token operator, Expr right",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

To weave the new expressions into the parser, we first change the parsing code\nfor assignment to call or().

\n
  private Expr assignment() {\n
lox/Parser.java
\nin assignment()
\nreplace 1 line
\n
    Expr expr = or();\n
\n\n    if (match(EQUAL)) {\n
\n
lox/Parser.java, in assignment(), replace 1 line
\n\n

The code to parse a series of or expressions mirrors other binary operators.

\n
lox/Parser.java
\nadd after assignment()
\n
  private Expr or() {\n    Expr expr = and();\n\n    while (match(OR)) {\n      Token operator = previous();\n      Expr right = and();\n      expr = new Expr.Logical(expr, operator, right);\n    }\n\n    return expr;\n  }\n
\n
lox/Parser.java, add after assignment()
\n\n

Its operands are the next higher level of precedence, the new and expression.

\n
lox/Parser.java
\nadd after or()
\n
  private Expr and() {\n    Expr expr = equality();\n\n    while (match(AND)) {\n      Token operator = previous();\n      Expr right = equality();\n      expr = new Expr.Logical(expr, operator, right);\n    }\n\n    return expr;\n  }\n
\n
lox/Parser.java, add after or()
\n\n

That calls equality() for its operands, and with that, the expression parser\nis all tied back together again. We’re ready to interpret.

\n
lox/Interpreter.java
\nadd after visitLiteralExpr()
\n
  @Override\n  public Object visitLogicalExpr(Expr.Logical expr) {\n    Object left = evaluate(expr.left);\n\n    if (expr.operator.type == TokenType.OR) {\n      if (isTruthy(left)) return left;\n    } else {\n      if (!isTruthy(left)) return left;\n    }\n\n    return evaluate(expr.right);\n  }\n
\n
lox/Interpreter.java, add after visitLiteralExpr()
\n\n

If you compare this to the earlier chapter’s visitBinaryExpr()\nmethod, you can see the difference. Here, we evaluate the left operand first. We\nlook at its value to see if we can short-circuit. If not, and only then, do we\nevaluate the right operand.

\n

The other interesting piece here is deciding what actual value to return. Since\nLox is dynamically typed, we allow operands of any type and use truthiness to\ndetermine what each operand represents. We apply similar reasoning to the\nresult. Instead of promising to literally return true or false, a logic\noperator merely guarantees it will return a value with appropriate truthiness.

\n

Fortunately, we have values with proper truthiness right at hand—the results\nof the operands themselves. So we use those. For example:

\n
print "hi" or 2; // "hi".\nprint nil or "yes"; // "yes".\n
\n

On the first line, \"hi\" is truthy, so the or short-circuits and returns\nthat. On the second line, nil is falsey, so it evaluates and returns the\nsecond operand, \"yes\".

\n

That covers all of the branching primitives in Lox. We’re ready to jump ahead to\nloops. You see what I did there? Jump. Ahead. Get it? See, it’s like a\nreference to . . . oh, forget it.

\n

9 . 4While Loops

\n

Lox features two looping control flow statements, while and for. The while\nloop is the simpler one, so we’ll start there. Its grammar is the same as in C.

\n
statement      → exprStmt\n               | ifStmt\n               | printStmt\n               | whileStmt\n               | block ;\n\nwhileStmt      → "while" "(" expression ")" statement ;\n
\n

We add another clause to the statement rule that points to the new rule for\nwhile. It takes a while keyword, followed by a parenthesized condition\nexpression, then a statement for the body. That new grammar rule gets a syntax tree node.

\n
      "Print      : Expr expression",\n
      "Var        : Token name, Expr initializer",\n
tool/GenerateAst.java
\nin main()
\nadd “,” to previous line
\n
      "While      : Expr condition, Stmt body"\n
    ));\n
\n
tool/GenerateAst.java, in main(), add “,” to previous line
\n\n\n

The node stores the condition and body. Here you can see why it’s nice to have\nseparate base classes for expressions and statements. The field declarations\nmake it clear that the condition is an expression and the body is a statement.

\n

Over in the parser, we follow the same process we used for if statements.\nFirst, we add another case in statement() to detect and match the leading\nkeyword.

\n
    if (match(PRINT)) return printStatement();\n
lox/Parser.java
\nin statement()
\n
    if (match(WHILE)) return whileStatement();\n
    if (match(LEFT_BRACE)) return new Stmt.Block(block());\n
\n
lox/Parser.java, in statement()
\n\n

That delegates the real work to this method:

\n
lox/Parser.java
\nadd after varDeclaration()
\n
  private Stmt whileStatement() {\n    consume(LEFT_PAREN, "Expect '(' after 'while'.");\n    Expr condition = expression();\n    consume(RIGHT_PAREN, "Expect ')' after condition.");\n    Stmt body = statement();\n\n    return new Stmt.While(condition, body);\n  }\n
\n
lox/Parser.java, add after varDeclaration()
\n\n

The grammar is dead simple and this is a straight translation of it to Java.\nSpeaking of translating straight to Java, here’s how we execute the new syntax:

\n
lox/Interpreter.java
\nadd after visitVarStmt()
\n
  @Override\n  public Void visitWhileStmt(Stmt.While stmt) {\n    while (isTruthy(evaluate(stmt.condition))) {\n      execute(stmt.body);\n    }\n    return null;\n  }\n
\n
lox/Interpreter.java, add after visitVarStmt()
\n\n

Like the visit method for if, this visitor uses the corresponding Java\nfeature. This method isn’t complex, but it makes Lox much more powerful. We can\nfinally write a program whose running time isn’t strictly bound by the length of\nthe source code.

\n

9 . 5For Loops

\n

We’re down to the last control flow construct, Ye Olde\nC-style for loop. I probably don’t need to remind you, but it looks like this:

\n
for (var i = 0; i < 10; i = i + 1) print i;\n
\n

In grammarese, that’s:

\n
statement      → exprStmt\n               | forStmt\n               | ifStmt\n               | printStmt\n               | whileStmt\n               | block ;\n\nforStmt        → "for" "(" ( varDecl | exprStmt | ";" )\n                 expression? ";"\n                 expression? ")" statement ;\n
\n\n

Inside the parentheses, you have three clauses separated by semicolons:

\n
    \n
  1. \n

    The first clause is the initializer. It is executed exactly once, before\nanything else. It’s usually an expression, but for convenience, we also\nallow a variable declaration. In that case, the variable is scoped to the\nrest of the for loop—the other two clauses and the body.

    \n
    2. \n
  2. \n

    Next is the condition. As in a while loop, this expression controls when\nto exit the loop. It’s evaluated once at the beginning of each iteration,\nincluding the first. If the result is truthy, it executes the loop body.\nOtherwise, it bails.

    \n
    3. \n
  3. \n

    The last clause is the increment. It’s an arbitrary expression that does\nsome work at the end of each loop iteration. The result of the expression is\ndiscarded, so it must have a side effect to be useful. In practice, it\nusually increments a variable.

    \n
    4. \n
\n

Any of these clauses can be omitted. Following the closing parenthesis is a\nstatement for the body, which is typically a block.

\n

9 . 5 . 1Desugaring

\n

That’s a lot of machinery, but note that none of it does anything you couldn’t\ndo with the statements we already have. If for loops didn’t support\ninitializer clauses, you could just put the initializer expression before the\nfor statement. Without an increment clause, you could simply put the increment\nexpression at the end of the body yourself.

\n

In other words, Lox doesn’t need for loops, they just make some common code\npatterns more pleasant to write. These kinds of features are called syntactic sugar. For example, the previous for loop\ncould be rewritten like so:

\n\n
{\n  var i = 0;\n  while (i < 10) {\n    print i;\n    i = i + 1;\n  }\n}\n
\n

This script has the exact same semantics as the previous one, though it’s not as\neasy on the eyes. Syntactic sugar features like Lox’s for loop make a language\nmore pleasant and productive to work in. But, especially in sophisticated\nlanguage implementations, every language feature that requires back-end support\nand optimization is expensive.

\n

We can have our cake and eat it too by desugaring. That funny word describes a process where\nthe front end takes code using syntax sugar and translates it to a more\nprimitive form that the back end already knows how to execute.

\n\n

We’re going to desugar for loops to the while loops and other statements the\ninterpreter already handles. In our simple interpreter, desugaring really\ndoesn’t save us much work, but it does give me an excuse to introduce you to the\ntechnique. So, unlike the previous statements, we won’t add a new syntax tree\nnode. Instead, we go straight to parsing. First, add an import we’ll need soon.

\n
import java.util.ArrayList;\n
lox/Parser.java
\n
import java.util.Arrays;\n
import java.util.List;\n
\n
lox/Parser.java
\n\n

Like every statement, we start parsing a for loop by matching its keyword.

\n
  private Stmt statement() {\n
lox/Parser.java
\nin statement()
\n
    if (match(FOR)) return forStatement();\n
    if (match(IF)) return ifStatement();\n
\n
lox/Parser.java, in statement()
\n\n

Here is where it gets interesting. The desugaring is going to happen here, so\nwe’ll build this method a piece at a time, starting with the opening parenthesis\nbefore the clauses.

\n
lox/Parser.java
\nadd after statement()
\n
  private Stmt forStatement() {\n    consume(LEFT_PAREN, "Expect '(' after 'for'.");\n\n    // More here...\n  }\n
\n
lox/Parser.java, add after statement()
\n\n

The first clause following that is the initializer.

\n
    consume(LEFT_PAREN, "Expect '(' after 'for'.");\n\n
lox/Parser.java
\nin forStatement()
\nreplace 1 line
\n
    Stmt initializer;\n    if (match(SEMICOLON)) {\n      initializer = null;\n    } else if (match(VAR)) {\n      initializer = varDeclaration();\n    } else {\n      initializer = expressionStatement();\n    }\n
  }\n
\n
lox/Parser.java, in forStatement(), replace 1 line
\n\n

If the token following the ( is a semicolon then the initializer has been\nomitted. Otherwise, we check for a var keyword to see if it’s a variable declaration. If neither of those matched, it\nmust be an expression. We parse that and wrap it in an expression statement so\nthat the initializer is always of type Stmt.

\n\n

Next up is the condition.

\n
      initializer = expressionStatement();\n    }\n
lox/Parser.java
\nin forStatement()
\n
\n\n    Expr condition = null;\n    if (!check(SEMICOLON)) {\n      condition = expression();\n    }\n    consume(SEMICOLON, "Expect ';' after loop condition.");\n
  }\n
\n
lox/Parser.java, in forStatement()
\n\n

Again, we look for a semicolon to see if the clause has been omitted. The last\nclause is the increment.

\n
    consume(SEMICOLON, "Expect ';' after loop condition.");\n
lox/Parser.java
\nin forStatement()
\n
\n\n    Expr increment = null;\n    if (!check(RIGHT_PAREN)) {\n      increment = expression();\n    }\n    consume(RIGHT_PAREN, "Expect ')' after for clauses.");\n
  }\n
\n
lox/Parser.java, in forStatement()
\n\n

It’s similar to the condition clause except this one is terminated by the\nclosing parenthesis. All that remains is the body.

\n\n
    consume(RIGHT_PAREN, "Expect ')' after for clauses.");\n
lox/Parser.java
\nin forStatement()
\n
    Stmt body = statement();\n\n    return body;\n
  }\n
\n
lox/Parser.java, in forStatement()
\n\n

We’ve parsed all of the various pieces of the for loop and the resulting AST\nnodes are sitting in a handful of Java local variables. This is where the\ndesugaring comes in. We take those and use them to synthesize syntax tree nodes\nthat express the semantics of the for loop, like the hand-desugared example I\nshowed you earlier.

\n

The code is a little simpler if we work backward, so we start with the increment\nclause.

\n
    Stmt body = statement();\n\n
lox/Parser.java
\nin forStatement()
\n
    if (increment != null) {\n      body = new Stmt.Block(\n          Arrays.asList(\n              body,\n              new Stmt.Expression(increment)));\n    }\n\n
    return body;\n
\n
lox/Parser.java, in forStatement()
\n\n

The increment, if there is one, executes after the body in each iteration of the\nloop. We do that by replacing the body with a little block that contains the\noriginal body followed by an expression statement that evaluates the increment.

\n
    }\n\n
lox/Parser.java
\nin forStatement()
\n
    if (condition == null) condition = new Expr.Literal(true);\n    body = new Stmt.While(condition, body);\n\n
    return body;\n
\n
lox/Parser.java, in forStatement()
\n\n

Next, we take the condition and the body and build the loop using a primitive\nwhile loop. If the condition is omitted, we jam in true to make an infinite\nloop.

\n
    body = new Stmt.While(condition, body);\n\n
lox/Parser.java
\nin forStatement()
\n
    if (initializer != null) {\n      body = new Stmt.Block(Arrays.asList(initializer, body));\n    }\n\n
    return body;\n
\n
lox/Parser.java, in forStatement()
\n\n

Finally, if there is an initializer, it runs once before the entire loop. We do\nthat by, again, replacing the whole statement with a block that runs the\ninitializer and then executes the loop.

\n

That’s it. Our interpreter now supports C-style for loops and we didn’t have\nto touch the Interpreter class at all. Since we desugared to nodes the\ninterpreter already knows how to visit, there is no more work to do.

\n

Finally, Lox is powerful enough to entertain us, at least for a few minutes.\nHere’s a tiny program to print the first 21 elements in the Fibonacci\nsequence:

\n
var a = 0;\nvar temp;\n\nfor (var b = 1; a < 10000; b = temp + b) {\n  print a;\n  temp = a;\n  a = b;\n}\n
\n
\n

Challenges

\n
    \n
  1. \n

    A few chapters from now, when Lox supports first-class functions and dynamic\ndispatch, we technically won’t need branching statements built into the\nlanguage. Show how conditional execution can be implemented in terms of\nthose. Name a language that uses this technique for its control flow.

    \n
    2. \n
  2. \n

    Likewise, looping can be implemented using those same tools, provided our\ninterpreter supports an important optimization. What is it, and why is it\nnecessary? Name a language that uses this technique for iteration.

    \n
    3. \n
  3. \n

    Unlike Lox, most other C-style languages also support break and continue\nstatements inside loops. Add support for break statements.

    \n

    The syntax is a break keyword followed by a semicolon. It should be a\nsyntax error to have a break statement appear outside of any enclosing\nloop. At runtime, a break statement causes execution to jump to the end of\nthe nearest enclosing loop and proceeds from there. Note that the break\nmay be nested inside other blocks and if statements that also need to be\nexited.

    \n
    4. \n
\n
\n
\n

Design Note: Spoonfuls of Syntactic Sugar

\n

When you design your own language, you choose how much syntactic sugar to pour\ninto the grammar. Do you make an unsweetened health food where each semantic\noperation maps to a single syntactic unit, or some decadent dessert where every\nbit of behavior can be expressed ten different ways? Successful languages\ninhabit all points along this continuum.

\n

On the extreme acrid end are those with ruthlessly minimal syntax like Lisp,\nForth, and Smalltalk. Lispers famously claim their language “has no syntax”,\nwhile Smalltalkers proudly show that you can fit the entire grammar on an index\ncard. This tribe has the philosophy that the language doesn’t need syntactic\nsugar. Instead, the minimal syntax and semantics it provides are powerful enough\nto let library code be as expressive as if it were part of the language itself.

\n

Near these are languages like C, Lua, and Go. They aim for simplicity and\nclarity over minimalism. Some, like Go, deliberately eschew both syntactic sugar\nand the kind of syntactic extensibility of the previous category. They want the\nsyntax to get out of the way of the semantics, so they focus on keeping both the\ngrammar and libraries simple. Code should be obvious more than beautiful.

\n

Somewhere in the middle you have languages like Java, C#, and Python. Eventually\nyou reach Ruby, C++, Perl, and D—languages which have stuffed so much syntax\ninto their grammar, they are running out of punctuation characters on the\nkeyboard.

\n

To some degree, location on the spectrum correlates with age. It’s relatively\neasy to add bits of syntactic sugar in later releases. New syntax is a crowd\npleaser, and it’s less likely to break existing programs than mucking with the\nsemantics. Once added, you can never take it away, so languages tend to sweeten\nwith time. One of the main benefits of creating a new language from scratch is\nit gives you an opportunity to scrape off those accumulated layers of frosting\nand start over.

\n

Syntactic sugar has a bad rap among the PL intelligentsia. There’s a real fetish\nfor minimalism in that crowd. There is some justification for that. Poorly\ndesigned, unneeded syntax raises the cognitive load without adding enough\nexpressiveness to carry its weight. Since there is always pressure to cram new\nfeatures into the language, it takes discipline and a focus on simplicity to\navoid bloat. Once you add some syntax, you’re stuck with it, so it’s smart to be\nparsimonious.

\n

At the same time, most successful languages do have fairly complex grammars, at\nleast by the time they are widely used. Programmers spend a ton of time in their\nlanguage of choice, and a few niceties here and there really can improve the\ncomfort and efficiency of their work.

\n

Striking the right balance—choosing the right level of sweetness for your\nlanguage—relies on your own sense of taste.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/dedication.html", "content": "\n\n\n\nDedication · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n\n\n
\n\n\n" }, { "path": "site/evaluating-expressions.html", "content": "\n\n\n\nEvaluating Expressions · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
7
\n

Evaluating Expressions

\n\n
\n

You are my creator, but I am your master; Obey!

\n

Mary Shelley, Frankenstein

\n
\n

If you want to properly set the mood for this chapter, try to conjure up a\nthunderstorm, one of those swirling tempests that likes to yank open shutters at\nthe climax of the story. Maybe toss in a few bolts of lightning. In this\nchapter, our interpreter will take breath, open its eyes, and execute some code.

\n

\"A\n\n

There are all manner of ways that language implementations make a computer do\nwhat the user’s source code commands. They can compile it to machine code,\ntranslate it to another high-level language, or reduce it to some bytecode\nformat for a virtual machine to run. For our first interpreter, though, we are\ngoing to take the simplest, shortest path and execute the syntax tree itself.

\n

Right now, our parser only supports expressions. So, to “execute” code, we will\nevaluate an expression and produce a value. For each kind of expression syntax\nwe can parse—literal, operator, etc.—we need a corresponding chunk of code\nthat knows how to evaluate that tree and produce a result. That raises two\nquestions:

\n
    \n
  1. \n

    What kinds of values do we produce?

    \n
    2. \n
  2. \n

    How do we organize those chunks of code?

    \n
    3. \n
\n

Taking them on one at a time . . . 

\n

7 . 1Representing Values

\n

In Lox, values are created by literals, computed by\nexpressions, and stored in variables. The user sees these as Lox objects, but\nthey are implemented in the underlying language our interpreter is written in.\nThat means bridging the lands of Lox’s dynamic typing and Java’s static types. A\nvariable in Lox can store a value of any (Lox) type, and can even store values\nof different types at different points in time. What Java type might we use to\nrepresent that?

\n\n

Given a Java variable with that static type, we must also be able to determine\nwhich kind of value it holds at runtime. When the interpreter executes a +\noperator, it needs to tell if it is adding two numbers or concatenating two\nstrings. Is there a Java type that can hold numbers, strings, Booleans, and\nmore? Is there one that can tell us what its runtime type is? There is! Good old\njava.lang.Object.

\n

In places in the interpreter where we need to store a Lox value, we can use\nObject as the type. Java has boxed versions of its primitive types that all\nsubclass Object, so we can use those for Lox’s built-in types:

\n\n\n \n \n\n\n\n\n \n \n\n\n \n \n\n\n \n \n\n\n \n \n\n\n \n \n\n\n
Lox typeJava representation
Any Lox valueObject
nilnull
BooleanBoolean
numberDouble
stringString
\n

Given a value of static type Object, we can determine if the runtime value is a\nnumber or a string or whatever using Java’s built-in instanceof operator. In\nother words, the JVM’s own object representation\nconveniently gives us everything we need to implement Lox’s built-in types.\nWe’ll have to do a little more work later when we add Lox’s notions of\nfunctions, classes, and instances, but Object and the boxed primitive classes\nare sufficient for the types we need right now.

\n\n

7 . 2Evaluating Expressions

\n

Next, we need blobs of code to implement the evaluation logic for each kind of\nexpression we can parse. We could stuff that code into the syntax tree classes\nin something like an interpret() method. In effect, we could tell each syntax\ntree node, “Interpret thyself”. This is the Gang of Four’s\nInterpreter design pattern. It’s a neat pattern, but like I mentioned\nearlier, it gets messy if we jam all sorts of logic into the tree classes.

\n

Instead, we’re going to reuse our groovy Visitor pattern. In the previous\nchapter, we created an AstPrinter class. It took in a syntax tree and\nrecursively traversed it, building up a string which it ultimately returned.\nThat’s almost exactly what a real interpreter does, except instead of\nconcatenating strings, it computes values.

\n

We start with a new class.

\n
lox/Interpreter.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nclass Interpreter implements Expr.Visitor<Object> {\n}\n
\n
lox/Interpreter.java, create new file
\n\n

The class declares that it’s a visitor. The return type of the visit methods\nwill be Object, the root class that we use to refer to a Lox value in our Java\ncode. To satisfy the Visitor interface, we need to define visit methods for each\nof the four expression tree classes our parser produces. We’ll start with the\nsimplest . . . 

\n

7 . 2 . 1Evaluating literals

\n

The leaves of an expression tree—the atomic bits of syntax that all other\nexpressions are composed of—are literals. Literals\nare almost values already, but the distinction is important. A literal is a bit\nof syntax that produces a value. A literal always appears somewhere in the\nuser’s source code. Lots of values are produced by computation and don’t exist\nanywhere in the code itself. Those aren’t literals. A literal comes from the\nparser’s domain. Values are an interpreter concept, part of the runtime’s world.

\n\n

So, much like we converted a literal token into a literal syntax tree node\nin the parser, now we convert the literal tree node into a runtime value. That\nturns out to be trivial.

\n
lox/Interpreter.java
\nin class Interpreter
\n
  @Override\n  public Object visitLiteralExpr(Expr.Literal expr) {\n    return expr.value;\n  }\n
\n
lox/Interpreter.java, in class Interpreter
\n\n

We eagerly produced the runtime value way back during scanning and stuffed it in\nthe token. The parser took that value and stuck it in the literal tree node,\nso to evaluate a literal, we simply pull it back out.

\n

7 . 2 . 2Evaluating parentheses

\n

The next simplest node to evaluate is grouping—the node you get as a result\nof using explicit parentheses in an expression.

\n
lox/Interpreter.java
\nin class Interpreter
\n
  @Override\n  public Object visitGroupingExpr(Expr.Grouping expr) {\n    return evaluate(expr.expression);\n  }\n
\n
lox/Interpreter.java, in class Interpreter
\n\n

A grouping node has a reference to an inner node\nfor the expression contained inside the parentheses. To evaluate the grouping\nexpression itself, we recursively evaluate that subexpression and return it.

\n

We rely on this helper method which simply sends the expression back into the\ninterpreter’s visitor implementation:

\n\n
lox/Interpreter.java
\nin class Interpreter
\n
  private Object evaluate(Expr expr) {\n    return expr.accept(this);\n  }\n
\n
lox/Interpreter.java, in class Interpreter
\n\n

7 . 2 . 3Evaluating unary expressions

\n

Like grouping, unary expressions have a single subexpression that we must\nevaluate first. The difference is that the unary expression itself does a little\nwork afterwards.

\n
lox/Interpreter.java
\nadd after visitLiteralExpr()
\n
  @Override\n  public Object visitUnaryExpr(Expr.Unary expr) {\n    Object right = evaluate(expr.right);\n\n    switch (expr.operator.type) {\n      case MINUS:\n        return -(double)right;\n    }\n\n    // Unreachable.\n    return null;\n  }\n
\n
lox/Interpreter.java, add after visitLiteralExpr()
\n\n

First, we evaluate the operand expression. Then we apply the unary operator\nitself to the result of that. There are two different unary expressions,\nidentified by the type of the operator token.

\n

Shown here is -, which negates the result of the subexpression. The\nsubexpression must be a number. Since we don’t statically know that in Java,\nwe cast it before performing the operation. This type\ncast happens at runtime when the - is evaluated. That’s the core of what makes\na language dynamically typed right there.

\n\n

You can start to see how evaluation recursively traverses the tree. We can’t\nevaluate the unary operator itself until after we evaluate its operand\nsubexpression. That means our interpreter is doing a post-order traversal—each node evaluates its children before doing its own work.

\n

The other unary operator is logical not.

\n
    switch (expr.operator.type) {\n
lox/Interpreter.java
\nin visitUnaryExpr()
\n
      case BANG:\n        return !isTruthy(right);\n
      case MINUS:\n
\n
lox/Interpreter.java, in visitUnaryExpr()
\n\n

The implementation is simple, but what is this “truthy” thing about? We need to\nmake a little side trip to one of the great questions of Western philosophy:\nWhat is truth?

\n

7 . 2 . 4Truthiness and falsiness

\n

OK, maybe we’re not going to really get into the universal question, but at\nleast inside the world of Lox, we need to decide what happens when you use\nsomething other than true or false in a logic operation like ! or any\nother place where a Boolean is expected.

\n

We could just say it’s an error because we don’t roll with implicit\nconversions, but most dynamically typed languages aren’t that ascetic. Instead,\nthey take the universe of values of all types and partition them into two sets,\none of which they define to be “true”, or “truthful”, or (my favorite) “truthy”,\nand the rest which are “false” or “falsey”. This partitioning is somewhat\narbitrary and gets weird in a few languages.

\n\n

Lox follows Ruby’s simple rule: false and nil are falsey, and everything else\nis truthy. We implement that like so:

\n
lox/Interpreter.java
\nadd after visitUnaryExpr()
\n
  private boolean isTruthy(Object object) {\n    if (object == null) return false;\n    if (object instanceof Boolean) return (boolean)object;\n    return true;\n  }\n
\n
lox/Interpreter.java, add after visitUnaryExpr()
\n\n

7 . 2 . 5Evaluating binary operators

\n

On to the last expression tree class, binary operators. There’s a handful of\nthem, and we’ll start with the arithmetic ones.

\n
lox/Interpreter.java
\nadd after evaluate()
\n
  @Override\n  public Object visitBinaryExpr(Expr.Binary expr) {\n    Object left = evaluate(expr.left);\n    Object right = evaluate(expr.right); \n\n    switch (expr.operator.type) {\n      case MINUS:\n        return (double)left - (double)right;\n      case SLASH:\n        return (double)left / (double)right;\n      case STAR:\n        return (double)left * (double)right;\n    }\n\n    // Unreachable.\n    return null;\n  }\n
\n
lox/Interpreter.java, add after evaluate()
\n\n\n

I think you can figure out what’s going on here. The main difference from the\nunary negation operator is that we have two operands to evaluate.

\n

I left out one arithmetic operator because it’s a little special.

\n
    switch (expr.operator.type) {\n      case MINUS:\n        return (double)left - (double)right;\n
lox/Interpreter.java
\nin visitBinaryExpr()
\n
      case PLUS:\n        if (left instanceof Double && right instanceof Double) {\n          return (double)left + (double)right;\n        } \n\n        if (left instanceof String && right instanceof String) {\n          return (String)left + (String)right;\n        }\n\n        break;\n
      case SLASH:\n
\n
lox/Interpreter.java, in visitBinaryExpr()
\n\n

The + operator can also be used to concatenate two strings. To handle that, we\ndon’t just assume the operands are a certain type and cast them, we\ndynamically check the type and choose the appropriate operation. This is why\nwe need our object representation to support instanceof.

\n\n

Next up are the comparison operators.

\n
    switch (expr.operator.type) {\n
lox/Interpreter.java
\nin visitBinaryExpr()
\n
      case GREATER:\n        return (double)left > (double)right;\n      case GREATER_EQUAL:\n        return (double)left >= (double)right;\n      case LESS:\n        return (double)left < (double)right;\n      case LESS_EQUAL:\n        return (double)left <= (double)right;\n
      case MINUS:\n
\n
lox/Interpreter.java, in visitBinaryExpr()
\n\n

They are basically the same as arithmetic. The only difference is that where the\narithmetic operators produce a value whose type is the same as the operands\n(numbers or strings), the comparison operators always produce a Boolean.

\n

The last pair of operators are equality.

\n
lox/Interpreter.java
\nin visitBinaryExpr()
\n
      case BANG_EQUAL: return !isEqual(left, right);\n      case EQUAL_EQUAL: return isEqual(left, right);\n
\n
lox/Interpreter.java, in visitBinaryExpr()
\n\n

Unlike the comparison operators which require numbers, the equality operators\nsupport operands of any type, even mixed ones. You can’t ask Lox if 3 is less\nthan \"three\", but you can ask if it’s equal to\nit.

\n\n

Like truthiness, the equality logic is hoisted out into a separate method.

\n
lox/Interpreter.java
\nadd after isTruthy()
\n
  private boolean isEqual(Object a, Object b) {\n    if (a == null && b == null) return true;\n    if (a == null) return false;\n\n    return a.equals(b);\n  }\n
\n
lox/Interpreter.java, add after isTruthy()
\n\n

This is one of those corners where the details of how we represent Lox objects\nin terms of Java matter. We need to correctly implement Lox’s notion of\nequality, which may be different from Java’s.

\n

Fortunately, the two are pretty similar. Lox doesn’t do implicit conversions in\nequality and Java does not either. We do have to handle nil/null specially\nso that we don’t throw a NullPointerException if we try to call equals() on\nnull. Otherwise, we’re fine. Java’s equals() method\non Boolean, Double, and String have the behavior we want for Lox.

\n\n

And that’s it! That’s all the code we need to correctly interpret a valid Lox\nexpression. But what about an invalid one? In particular, what happens when a\nsubexpression evaluates to an object of the wrong type for the operation being\nperformed?

\n

7 . 3Runtime Errors

\n

I was cavalier about jamming casts in whenever a subexpression produces an\nObject and the operator requires it to be a number or a string. Those casts can\nfail. Even though the user’s code is erroneous, if we want to make a usable language, we are responsible for handling that error\ngracefully.

\n\n

It’s time for us to talk about runtime errors. I spilled a lot of ink in the\nprevious chapters talking about error handling, but those were all syntax or\nstatic errors. Those are detected and reported before any code is executed.\nRuntime errors are failures that the language semantics demand we detect and\nreport while the program is running (hence the name).

\n

Right now, if an operand is the wrong type for the operation being performed,\nthe Java cast will fail and the JVM will throw a ClassCastException. That\nunwinds the whole stack and exits the application, vomiting a Java stack trace\nonto the user. That’s probably not what we want. The fact that Lox is\nimplemented in Java should be a detail hidden from the user. Instead, we want\nthem to understand that a Lox runtime error occurred, and give them an error\nmessage relevant to our language and their program.

\n

The Java behavior does have one thing going for it, though. It correctly stops\nexecuting any code when the error occurs. Let’s say the user enters some\nexpression like:

\n
2 * (3 / -"muffin")\n
\n

You can’t negate a muffin, so we need to report a\nruntime error at that inner - expression. That in turn means we can’t evaluate\nthe / expression since it has no meaningful right operand. Likewise for the\n*. So when a runtime error occurs deep in some expression, we need to escape\nall the way out.

\n\n

We could print a runtime error and then abort the process and exit the\napplication entirely. That has a certain melodramatic flair. Sort of the\nprogramming language interpreter equivalent of a mic drop.

\n

Tempting as that is, we should probably do something a little less cataclysmic.\nWhile a runtime error needs to stop evaluating the expression, it shouldn’t\nkill the interpreter. If a user is running the REPL and has a typo in a line\nof code, they should still be able to keep the session going and enter more code\nafter that.

\n

7 . 3 . 1Detecting runtime errors

\n

Our tree-walk interpreter evaluates nested expressions using recursive method\ncalls, and we need to unwind out of all of those. Throwing an exception in Java\nis a fine way to accomplish that. However, instead of using Java’s own cast\nfailure, we’ll define a Lox-specific one so that we can handle it how we want.

\n

Before we do the cast, we check the object’s type ourselves. So, for unary -,\nwe add:

\n
      case MINUS:\n
lox/Interpreter.java
\nin visitUnaryExpr()
\n
        checkNumberOperand(expr.operator, right);\n
        return -(double)right;\n
\n
lox/Interpreter.java, in visitUnaryExpr()
\n\n

The code to check the operand is:

\n
lox/Interpreter.java
\nadd after visitUnaryExpr()
\n
  private void checkNumberOperand(Token operator, Object operand) {\n    if (operand instanceof Double) return;\n    throw new RuntimeError(operator, "Operand must be a number.");\n  }\n
\n
lox/Interpreter.java, add after visitUnaryExpr()
\n\n

When the check fails, it throws one of these:

\n
lox/RuntimeError.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nclass RuntimeError extends RuntimeException {\n  final Token token;\n\n  RuntimeError(Token token, String message) {\n    super(message);\n    this.token = token;\n  }\n}\n
\n
lox/RuntimeError.java, create new file
\n\n

Unlike the Java cast exception, our class tracks the\ntoken that identifies where in the user’s code the runtime error came from. As\nwith static errors, this helps the user know where to fix their code.

\n\n

We need similar checking for the binary operators. Since I promised you every\nsingle line of code needed to implement the interpreters, I’ll run through them\nall.

\n

Greater than:

\n
      case GREATER:\n
lox/Interpreter.java
\nin visitBinaryExpr()
\n
        checkNumberOperands(expr.operator, left, right);\n
        return (double)left > (double)right;\n
\n
lox/Interpreter.java, in visitBinaryExpr()
\n\n

Greater than or equal to:

\n
      case GREATER_EQUAL:\n
lox/Interpreter.java
\nin visitBinaryExpr()
\n
        checkNumberOperands(expr.operator, left, right);\n
        return (double)left >= (double)right;\n
\n
lox/Interpreter.java, in visitBinaryExpr()
\n\n

Less than:

\n
      case LESS:\n
lox/Interpreter.java
\nin visitBinaryExpr()
\n
        checkNumberOperands(expr.operator, left, right);\n
        return (double)left < (double)right;\n
\n
lox/Interpreter.java, in visitBinaryExpr()
\n\n

Less than or equal to:

\n
      case LESS_EQUAL:\n
lox/Interpreter.java
\nin visitBinaryExpr()
\n
        checkNumberOperands(expr.operator, left, right);\n
        return (double)left <= (double)right;\n
\n
lox/Interpreter.java, in visitBinaryExpr()
\n\n

Subtraction:

\n
      case MINUS:\n
lox/Interpreter.java
\nin visitBinaryExpr()
\n
        checkNumberOperands(expr.operator, left, right);\n
        return (double)left - (double)right;\n
\n
lox/Interpreter.java, in visitBinaryExpr()
\n\n

Division:

\n
      case SLASH:\n
lox/Interpreter.java
\nin visitBinaryExpr()
\n
        checkNumberOperands(expr.operator, left, right);\n
        return (double)left / (double)right;\n
\n
lox/Interpreter.java, in visitBinaryExpr()
\n\n

Multiplication:

\n
      case STAR:\n
lox/Interpreter.java
\nin visitBinaryExpr()
\n
        checkNumberOperands(expr.operator, left, right);\n
        return (double)left * (double)right;\n
\n
lox/Interpreter.java, in visitBinaryExpr()
\n\n

All of those rely on this validator, which is virtually the same as the unary\none:

\n
lox/Interpreter.java
\nadd after checkNumberOperand()
\n
  private void checkNumberOperands(Token operator,\n                                   Object left, Object right) {\n    if (left instanceof Double && right instanceof Double) return;\n    \n    throw new RuntimeError(operator, "Operands must be numbers.");\n  }\n
\n
lox/Interpreter.java, add after checkNumberOperand()
\n\n\n

The last remaining operator, again the odd one out, is addition. Since + is\noverloaded for numbers and strings, it already has code to check the types. All\nwe need to do is fail if neither of the two success cases match.

\n
          return (String)left + (String)right;\n        }\n\n
lox/Interpreter.java
\nin visitBinaryExpr()
\nreplace 1 line
\n
        throw new RuntimeError(expr.operator,\n            "Operands must be two numbers or two strings.");\n
      case SLASH:\n
\n
lox/Interpreter.java, in visitBinaryExpr(), replace 1 line
\n\n

That gets us detecting runtime errors deep in the innards of the evaluator. The\nerrors are getting thrown. The next step is to write the code that catches them.\nFor that, we need to wire up the Interpreter class into the main Lox class that\ndrives it.

\n

7 . 4Hooking Up the Interpreter

\n

The visit methods are sort of the guts of the Interpreter class, where the real\nwork happens. We need to wrap a skin around them to interface with the rest of\nthe program. The Interpreter’s public API is simply one method.

\n
lox/Interpreter.java
\nin class Interpreter
\n
  void interpret(Expr expression) { \n    try {\n      Object value = evaluate(expression);\n      System.out.println(stringify(value));\n    } catch (RuntimeError error) {\n      Lox.runtimeError(error);\n    }\n  }\n
\n
lox/Interpreter.java, in class Interpreter
\n\n

This takes in a syntax tree for an expression and evaluates it. If that\nsucceeds, evaluate() returns an object for the result value. interpret()\nconverts that to a string and shows it to the user. To convert a Lox value to a\nstring, we rely on:

\n
lox/Interpreter.java
\nadd after isEqual()
\n
  private String stringify(Object object) {\n    if (object == null) return "nil";\n\n    if (object instanceof Double) {\n      String text = object.toString();\n      if (text.endsWith(".0")) {\n        text = text.substring(0, text.length() - 2);\n      }\n      return text;\n    }\n\n    return object.toString();\n  }\n
\n
lox/Interpreter.java, add after isEqual()
\n\n

This is another of those pieces of code like isTruthy() that crosses the\nmembrane between the user’s view of Lox objects and their internal\nrepresentation in Java.

\n

It’s pretty straightforward. Since Lox was designed to be familiar to someone\ncoming from Java, things like Booleans look the same in both languages. The two\nedge cases are nil, which we represent using Java’s null, and numbers.

\n

Lox uses double-precision numbers even for integer values. In that case, they\nshould print without a decimal point. Since Java has both floating point and\ninteger types, it wants you to know which one you’re using. It tells you by\nadding an explicit .0 to integer-valued doubles. We don’t care about that, so\nwe hack it off the end.

\n\n

7 . 4 . 1Reporting runtime errors

\n

If a runtime error is thrown while evaluating the expression, interpret()\ncatches it. This lets us report the error to the user and then gracefully\ncontinue. All of our existing error reporting code lives in the Lox class, so we\nput this method there too:

\n
lox/Lox.java
\nadd after error()
\n
  static void runtimeError(RuntimeError error) {\n    System.err.println(error.getMessage() +\n        "\\n[line " + error.token.line + "]");\n    hadRuntimeError = true;\n  }\n
\n
lox/Lox.java, add after error()
\n\n

We use the token associated with the RuntimeError to tell the user what line of\ncode was executing when the error occurred. Even better would be to give the\nuser an entire call stack to show how they got to be executing that code. But\nwe don’t have function calls yet, so I guess we don’t have to worry about it.

\n

After showing the error, runtimeError() sets this field:

\n
  static boolean hadError = false;\n
lox/Lox.java
\nin class Lox
\n
  static boolean hadRuntimeError = false;\n\n
  public static void main(String[] args) throws IOException {\n
\n
lox/Lox.java, in class Lox
\n\n

That field plays a small but important role.

\n
    run(new String(bytes, Charset.defaultCharset()));\n\n    // Indicate an error in the exit code.\n    if (hadError) System.exit(65);\n
lox/Lox.java
\nin runFile()
\n
    if (hadRuntimeError) System.exit(70);\n
  }\n
\n
lox/Lox.java, in runFile()
\n\n

If the user is running a Lox script from a file and a\nruntime error occurs, we set an exit code when the process quits to let the\ncalling process know. Not everyone cares about shell etiquette, but we do.

\n\n

7 . 4 . 2Running the interpreter

\n

Now that we have an interpreter, the Lox class can start using it.

\n
public class Lox {\n
lox/Lox.java
\nin class Lox
\n
  private static final Interpreter interpreter = new Interpreter();\n
  static boolean hadError = false;\n
\n
lox/Lox.java, in class Lox
\n\n

We make the field static so that successive calls to run() inside a REPL\nsession reuse the same interpreter. That doesn’t make a difference now, but it\nwill later when the interpreter stores global variables. Those variables should\npersist throughout the REPL session.

\n

Finally, we remove the line of temporary code from the last chapter for\nprinting the syntax tree and replace it with this:

\n
    // Stop if there was a syntax error.\n    if (hadError) return;\n\n
lox/Lox.java
\nin run()
\nreplace 1 line
\n
    interpreter.interpret(expression);\n
  }\n
\n
lox/Lox.java, in run(), replace 1 line
\n\n

We have an entire language pipeline now: scanning, parsing, and\nexecution. Congratulations, you now have your very own arithmetic calculator.

\n

As you can see, the interpreter is pretty bare bones. But the Interpreter class\nand the Visitor pattern we’ve set up today form the skeleton that later chapters\nwill stuff full of interesting guts—variables, functions, etc. Right now, the\ninterpreter doesn’t do very much, but it’s alive!

\"A\n
\n

Challenges

\n
    \n
  1. \n

    Allowing comparisons on types other than numbers could be useful. The\noperators might have a reasonable interpretation for strings. Even\ncomparisons among mixed types, like 3 < \"pancake\" could be handy to enable\nthings like ordered collections of heterogeneous types. Or it could simply\nlead to bugs and confusion.

    \n

    Would you extend Lox to support comparing other types? If so, which pairs of\ntypes do you allow and how do you define their ordering? Justify your\nchoices and compare them to other languages.

    \n
    2. \n
  2. \n

    Many languages define + such that if either operand is a string, the\nother is converted to a string and the results are then concatenated. For\nexample, \"scone\" + 4 would yield scone4. Extend the code in\nvisitBinaryExpr() to support that.

    \n
    3. \n
  3. \n

    What happens right now if you divide a number by zero? What do you think\nshould happen? Justify your choice. How do other languages you know handle\ndivision by zero, and why do they make the choices they do?

    \n

    Change the implementation in visitBinaryExpr() to detect and report a\nruntime error for this case.

    \n
    4. \n
\n
\n
\n

Design Note: Static and Dynamic Typing

\n

Some languages, like Java, are statically typed which means type errors are\ndetected and reported at compile time before any code is run. Others, like Lox,\nare dynamically typed and defer checking for type errors until runtime right\nbefore an operation is attempted. We tend to consider this a black-and-white\nchoice, but there is actually a continuum between them.

\n

It turns out even most statically typed languages do some type checks at\nruntime. The type system checks most type rules statically, but inserts runtime\nchecks in the generated code for other operations.

\n

For example, in Java, the static type system assumes a cast expression will\nalways safely succeed. After you cast some value, you can statically treat it as\nthe destination type and not get any compile errors. But downcasts can fail,\nobviously. The only reason the static checker can presume that casts always\nsucceed without violating the language’s soundness guarantees, is because the\ncast is checked at runtime and throws an exception on failure.

\n

A more subtle example is covariant arrays in Java and C#. The static\nsubtyping rules for arrays allow operations that are not sound. Consider:

\n
Object[] stuff = new Integer[1];\nstuff[0] = "not an int!";\n
\n

This code compiles without any errors. The first line upcasts the Integer array\nand stores it in a variable of type Object array. The second line stores a\nstring in one of its cells. The Object array type statically allows that—strings are Objects—but the actual Integer array that stuff refers to\nat runtime should never have a string in it! To avoid that catastrophe, when you\nstore a value in an array, the JVM does a runtime check to make sure it’s an\nallowed type. If not, it throws an ArrayStoreException.

\n

Java could have avoided the need to check this at runtime by disallowing the\ncast on the first line. It could make arrays invariant such that an array of\nIntegers is not an array of Objects. That’s statically sound, but it prohibits\ncommon and safe patterns of code that only read from arrays. Covariance is safe\nif you never write to the array. Those patterns were particularly important\nfor usability in Java 1.0 before it supported generics. James Gosling and the\nother Java designers traded off a little static safety and performance—those\narray store checks take time—in return for some flexibility.

\n

There are few modern statically typed languages that don’t make that trade-off\nsomewhere. Even Haskell will let you run code with non-exhaustive matches. If\nyou find yourself designing a statically typed language, keep in mind that you\ncan sometimes give users more flexibility without sacrificing too many of the\nbenefits of static safety by deferring some type checks until runtime.

\n

On the other hand, a key reason users choose statically typed languages is\nbecause of the confidence the language gives them that certain kinds of errors\ncan never occur when their program is run. Defer too many type checks until\nruntime, and you erode that confidence.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/functions.html", "content": "\n\n\n\nFunctions · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
10
\n

Functions

\n\n
\n

And that is also the way the human mind works—by the compounding of old\nideas into new structures that become new ideas that can themselves be used in\ncompounds, and round and round endlessly, growing ever more remote from the\nbasic earthbound imagery that is each language’s soil.

\n

Douglas R. Hofstadter, I Am a Strange Loop

\n
\n

This chapter marks the culmination of a lot of hard work. The previous chapters\nadd useful functionality in their own right, but each also supplies a piece of a\npuzzle. We’ll take those pieces—expressions,\nstatements, variables, control flow, and lexical scope—add a couple more, and\nassemble them all into support for real user-defined functions and function\ncalls.

\n\n

10 . 1Function Calls

\n

You’re certainly familiar with C-style function call syntax, but the grammar is\nmore subtle than you may realize. Calls are typically to named functions like:

\n
average(1, 2);\n
\n

But the name of the function being called isn’t\nactually part of the call syntax. The thing being called—the callee—can be any expression that evaluates to a function. (Well, it does have to be a\npretty high precedence expression, but parentheses take care of that.) For\nexample:

\n\n
getCallback()();\n
\n

There are two call expressions here. The first pair of parentheses has\ngetCallback as its callee. But the second call has the entire getCallback()\nexpression as its callee. It is the parentheses following an expression that\nindicate a function call. You can think of a call as sort of like a postfix\noperator that starts with (.

\n

This “operator” has higher precedence than any other operator, even the unary\nones. So we slot it into the grammar by having the unary rule bubble up to a\nnew call rule.

\n

\n
unary          → ( "!" | "-" ) unary | call ;\ncall           → primary ( "(" arguments? ")" )* ;\n
\n

This rule matches a primary expression followed by zero or more function calls.\nIf there are no parentheses, this parses a bare primary expression. Otherwise,\neach call is recognized by a pair of parentheses with an optional list of\narguments inside. The argument list grammar is:

\n\n
arguments      → expression ( "," expression )* ;\n
\n

This rule requires at least one argument expression, followed by zero or more\nother expressions, each preceded by a comma. To handle zero-argument calls, the\ncall rule itself considers the entire arguments production to be optional.

\n

I admit, this seems more grammatically awkward than you’d expect for the\nincredibly common “zero or more comma-separated things” pattern. There are some\nsophisticated metasyntaxes that handle this better, but in our BNF and in many\nlanguage specs I’ve seen, it is this cumbersome.

\n

Over in our syntax tree generator, we add a new\nnode.

\n
      "Binary   : Expr left, Token operator, Expr right",\n
tool/GenerateAst.java
\nin main()
\n
      "Call     : Expr callee, Token paren, List<Expr> arguments",\n
      "Grouping : Expr expression",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

It stores the callee expression and a list of expressions for the arguments. It\nalso stores the token for the closing parenthesis. We’ll use that token’s\nlocation when we report a runtime error caused by a function call.

\n

Crack open the parser. Where unary() used to jump straight to primary(),\nchange it to call, well, call().

\n
      return new Expr.Unary(operator, right);\n    }\n\n
lox/Parser.java
\nin unary()
\nreplace 1 line
\n
    return call();\n
  }\n
\n
lox/Parser.java, in unary(), replace 1 line
\n\n

Its definition is:

\n
lox/Parser.java
\nadd after unary()
\n
  private Expr call() {\n    Expr expr = primary();\n\n    while (true) { \n      if (match(LEFT_PAREN)) {\n        expr = finishCall(expr);\n      } else {\n        break;\n      }\n    }\n\n    return expr;\n  }\n
\n
lox/Parser.java, add after unary()
\n\n

The code here doesn’t quite line up with the grammar rules. I moved a few things\naround to make the code cleaner—one of the luxuries we have with a\nhandwritten parser. But it’s roughly similar to how we parse infix operators.\nFirst, we parse a primary expression, the “left operand” to the call. Then, each\ntime we see a (, we call finishCall() to parse the call expression using the\npreviously parsed expression as the callee. The returned expression becomes the\nnew expr and we loop to see if the result is itself called.

\n\n

The code to parse the argument list is in this helper:

\n
lox/Parser.java
\nadd after unary()
\n
  private Expr finishCall(Expr callee) {\n    List<Expr> arguments = new ArrayList<>();\n    if (!check(RIGHT_PAREN)) {\n      do {\n        arguments.add(expression());\n      } while (match(COMMA));\n    }\n\n    Token paren = consume(RIGHT_PAREN,\n                          "Expect ')' after arguments.");\n\n    return new Expr.Call(callee, paren, arguments);\n  }\n
\n
lox/Parser.java, add after unary()
\n\n

This is more or less the arguments grammar rule translated to code, except\nthat we also handle the zero-argument case. We check for that case first by\nseeing if the next token is ). If it is, we don’t try to parse any arguments.

\n

Otherwise, we parse an expression, then look for a comma indicating that there\nis another argument after that. We keep doing that as long as we find commas\nafter each expression. When we don’t find a comma, then the argument list must\nbe done and we consume the expected closing parenthesis. Finally, we wrap the\ncallee and those arguments up into a call AST node.

\n

10 . 1 . 1Maximum argument counts

\n

Right now, the loop where we parse arguments has no bound. If you want to call a\nfunction and pass a million arguments to it, the parser would have no problem\nwith it. Do we want to limit that?

\n

Other languages have various approaches. The C standard says a conforming\nimplementation has to support at least 127 arguments to a function, but\ndoesn’t say there’s any upper limit. The Java specification says a method can\naccept no more than 255 arguments.

\n\n

Our Java interpreter for Lox doesn’t really need a limit, but having a maximum\nnumber of arguments will simplify our bytecode interpreter in Part III. We\nwant our two interpreters to be compatible with each other, even in weird corner\ncases like this, so we’ll add the same limit to jlox.

\n
      do {\n
lox/Parser.java
\nin finishCall()
\n
        if (arguments.size() >= 255) {\n          error(peek(), "Can't have more than 255 arguments.");\n        }\n
        arguments.add(expression());\n
\n
lox/Parser.java, in finishCall()
\n\n

Note that the code here reports an error if it encounters too many arguments,\nbut it doesn’t throw the error. Throwing is how we kick into panic mode which\nis what we want if the parser is in a confused state and doesn’t know where it\nis in the grammar anymore. But here, the parser is still in a perfectly valid\nstate—it just found too many arguments. So it reports the error and keeps on\nkeepin’ on.

\n

10 . 1 . 2Interpreting function calls

\n

We don’t have any functions we can call, so it seems weird to start implementing\ncalls first, but we’ll worry about that when we get there. First, our\ninterpreter needs a new import.

\n
lox/Interpreter.java
\n
import java.util.ArrayList;\n
import java.util.List;\n
\n
lox/Interpreter.java
\n\n

As always, interpretation starts with a new visit method for our new call\nexpression node.

\n
lox/Interpreter.java
\nadd after visitBinaryExpr()
\n
  @Override\n  public Object visitCallExpr(Expr.Call expr) {\n    Object callee = evaluate(expr.callee);\n\n    List<Object> arguments = new ArrayList<>();\n    for (Expr argument : expr.arguments) { \n      arguments.add(evaluate(argument));\n    }\n\n    LoxCallable function = (LoxCallable)callee;\n    return function.call(this, arguments);\n  }\n
\n
lox/Interpreter.java, add after visitBinaryExpr()
\n\n

First, we evaluate the expression for the callee. Typically, this expression is\njust an identifier that looks up the function by its name, but it could be\nanything. Then we evaluate each of the argument expressions in order and store\nthe resulting values in a list.

\n\n

Once we’ve got the callee and the arguments ready, all that remains is to\nperform the call. We do that by casting the callee to a LoxCallable and then invoking a call() method on it.\nThe Java representation of any Lox object that can be called like a function\nwill implement this interface. That includes user-defined functions, naturally,\nbut also class objects since classes are “called” to construct new instances.\nWe’ll also use it for one more purpose shortly.

\n\n

There isn’t too much to this new interface.

\n
lox/LoxCallable.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.util.List;\n\ninterface LoxCallable {\n  Object call(Interpreter interpreter, List<Object> arguments);\n}\n
\n
lox/LoxCallable.java, create new file
\n\n

We pass in the interpreter in case the class implementing call() needs it. We\nalso give it the list of evaluated argument values. The implementer’s job is\nthen to return the value that the call expression produces.

\n

10 . 1 . 3Call type errors

\n

Before we get to implementing LoxCallable, we need to make the visit method a\nlittle more robust. It currently ignores a couple of failure modes that we can’t\npretend won’t occur. First, what happens if the callee isn’t actually something\nyou can call? What if you try to do this:

\n
"totally not a function"();\n
\n

Strings aren’t callable in Lox. The runtime representation of a Lox string is a\nJava string, so when we cast that to LoxCallable, the JVM will throw a\nClassCastException. We don’t want our interpreter to vomit out some nasty Java\nstack trace and die. Instead, we need to check the type ourselves first.

\n
    }\n\n
lox/Interpreter.java
\nin visitCallExpr()
\n
    if (!(callee instanceof LoxCallable)) {\n      throw new RuntimeError(expr.paren,\n          "Can only call functions and classes.");\n    }\n\n
    LoxCallable function = (LoxCallable)callee;\n
\n
lox/Interpreter.java, in visitCallExpr()
\n\n

We still throw an exception, but now we’re throwing our own exception type, one\nthat the interpreter knows to catch and report gracefully.

\n

10 . 1 . 4Checking arity

\n

The other problem relates to the function’s arity. Arity is the fancy term\nfor the number of arguments a function or operation expects. Unary operators\nhave arity one, binary operators two, etc. With functions, the arity is\ndetermined by the number of parameters it declares.

\n
fun add(a, b, c) {\n  print a + b + c;\n}\n
\n

This function defines three parameters, a, b, and c, so its arity is\nthree and it expects three arguments. So what if you try to call it like this:

\n
add(1, 2, 3, 4); // Too many.\nadd(1, 2);       // Too few.\n
\n

Different languages take different approaches to this problem. Of course, most\nstatically typed languages check this at compile time and refuse to compile the\ncode if the argument count doesn’t match the function’s arity. JavaScript\ndiscards any extra arguments you pass. If you don’t pass enough, it fills in the\nmissing parameters with the magic sort-of-like-null-but-not-really value\nundefined. Python is stricter. It raises a runtime error if the argument list\nis too short or too long.

\n

I think the latter is a better approach. Passing the wrong number of arguments\nis almost always a bug, and it’s a mistake I do make in practice. Given that,\nthe sooner the implementation draws my attention to it, the better. So for Lox,\nwe’ll take Python’s approach. Before invoking the callable, we check to see if\nthe argument list’s length matches the callable’s arity.

\n
    LoxCallable function = (LoxCallable)callee;\n
lox/Interpreter.java
\nin visitCallExpr()
\n
    if (arguments.size() != function.arity()) {\n      throw new RuntimeError(expr.paren, "Expected " +\n          function.arity() + " arguments but got " +\n          arguments.size() + ".");\n    }\n\n
    return function.call(this, arguments);\n
\n
lox/Interpreter.java, in visitCallExpr()
\n\n

That requires a new method on the LoxCallable interface to ask it its arity.

\n
interface LoxCallable {\n
lox/LoxCallable.java
\nin interface LoxCallable
\n
  int arity();\n
  Object call(Interpreter interpreter, List<Object> arguments);\n
\n
lox/LoxCallable.java, in interface LoxCallable
\n\n

We could push the arity checking into the concrete implementation of call().\nBut, since we’ll have multiple classes implementing LoxCallable, that would end\nup with redundant validation spread across a few classes. Hoisting it up into\nthe visit method lets us do it in one place.

\n

10 . 2Native Functions

\n

We can theoretically call functions, but we have no functions to call yet.\nBefore we get to user-defined functions, now is a good time to introduce a vital\nbut often overlooked facet of language implementations—native functions. These are functions that the\ninterpreter exposes to user code but that are implemented in the host language\n(in our case Java), not the language being implemented (Lox).

\n

Sometimes these are called primitives, external functions, or foreign\nfunctions. Since these functions can be called while the user’s program is\nrunning, they form part of the implementation’s runtime. A lot of programming\nlanguage books gloss over these because they aren’t conceptually interesting.\nThey’re mostly grunt work.

\n\n

But when it comes to making your language actually good at doing useful stuff,\nthe native functions your implementation provides are key. They provide access\nto the fundamental services that all programs are defined in terms of. If you\ndon’t provide native functions to access the file system, a user’s going to have\na hell of a time writing a program that reads and displays a file.

\n\n

Many languages also allow users to provide their own native functions. The\nmechanism for doing so is called a foreign function interface (FFI),\nnative extension, native interface, or something along those lines.\nThese are nice because they free the language implementer from providing access\nto every single capability the underlying platform supports. We won’t define an\nFFI for jlox, but we will add one native function to give you an idea of what it\nlooks like.

\n

10 . 2 . 1Telling time

\n

When we get to Part III and start working on a much more efficient\nimplementation of Lox, we’re going to care deeply about performance. Performance\nwork requires measurement, and that in turn means benchmarks. These are\nprograms that measure the time it takes to exercise some corner of the\ninterpreter.

\n

We could measure the time it takes to start up the interpreter, run the\nbenchmark, and exit, but that adds a lot of overhead—JVM startup time, OS\nshenanigans, etc. That stuff does matter, of course, but if you’re just trying\nto validate an optimization to some piece of the interpreter, you don’t want\nthat overhead obscuring your results.

\n

A nicer solution is to have the benchmark script itself measure the time elapsed\nbetween two points in the code. To do that, a Lox program needs to be able to\ntell time. There’s no way to do that now—you can’t implement a useful clock\n“from scratch” without access to the underlying clock on the computer.

\n

So we’ll add clock(), a native function that returns the number of seconds\nthat have passed since some fixed point in time. The difference between two\nsuccessive invocations tells you how much time elapsed between the two calls.\nThis function is defined in the global scope, so let’s ensure the interpreter\nhas access to that.

\n
class Interpreter implements Expr.Visitor<Object>,\n                             Stmt.Visitor<Void> {\n
lox/Interpreter.java
\nin class Interpreter
\nreplace 1 line
\n
  final Environment globals = new Environment();\n  private Environment environment = globals;\n
\n\n  void interpret(List<Stmt> statements) {\n
\n
lox/Interpreter.java, in class Interpreter, replace 1 line
\n\n

The environment field in the interpreter changes as we enter and exit local\nscopes. It tracks the current environment. This new globals field holds a\nfixed reference to the outermost global environment.

\n

When we instantiate an Interpreter, we stuff the native function in that global\nscope.

\n
  private Environment environment = globals;\n\n
lox/Interpreter.java
\nin class Interpreter
\n
  Interpreter() {\n    globals.define("clock", new LoxCallable() {\n      @Override\n      public int arity() { return 0; }\n\n      @Override\n      public Object call(Interpreter interpreter,\n                         List<Object> arguments) {\n        return (double)System.currentTimeMillis() / 1000.0;\n      }\n\n      @Override\n      public String toString() { return "<native fn>"; }\n    });\n  }\n\n
  void interpret(List<Stmt> statements) {\n
\n
lox/Interpreter.java, in class Interpreter
\n\n

This defines a variable named “clock”. Its value is a\nJava anonymous class that implements LoxCallable. The clock() function takes\nno arguments, so its arity is zero. The implementation of call() calls the\ncorresponding Java function and converts the result to a double value in\nseconds.

\n\n

If we wanted to add other native functions—reading input from the user,\nworking with files, etc.—we could add them each as their own anonymous class\nthat implements LoxCallable. But for the book, this one is really all we need.

\n

Let’s get ourselves out of the function-defining business and let our users\ntake over . . . 

\n

10 . 3Function Declarations

\n

We finally get to add a new production to the declaration rule we introduced\nback when we added variables. Function declarations, like variables, bind a new\nname. That means they are allowed only in places where\na declaration is permitted.

\n\n
declaration    → funDecl\n               | varDecl\n               | statement ;\n
\n

The updated declaration rule references this new rule:

\n
funDecl        → "fun" function ;\nfunction       → IDENTIFIER "(" parameters? ")" block ;\n
\n

The main funDecl rule uses a separate helper rule function. A function\ndeclaration statement is the fun keyword followed by the actual function-y\nstuff. When we get to classes, we’ll reuse that function rule for declaring\nmethods. Those look similar to function declarations, but aren’t preceded by\nfun.

\n\n

The function itself is a name followed by the parenthesized parameter list and\nthe body. The body is always a braced block, using the same grammar rule that\nblock statements use. The parameter list uses this rule:

\n
parameters     → IDENTIFIER ( "," IDENTIFIER )* ;\n
\n

It’s like the earlier arguments rule, except that each parameter is an\nidentifier, not an expression. That’s a lot of new syntax for the parser to chew\nthrough, but the resulting AST node isn’t too bad.

\n
      "Expression : Expr expression",\n
tool/GenerateAst.java
\nin main()
\n
      "Function   : Token name, List<Token> params," +\n                  " List<Stmt> body",\n
      "If         : Expr condition, Stmt thenBranch," +\n
\n
tool/GenerateAst.java, in main()
\n\n\n

A function node has a name, a list of parameters (their names), and then the\nbody. We store the body as the list of statements contained inside the curly\nbraces.

\n

Over in the parser, we weave in the new declaration.

\n
    try {\n
lox/Parser.java
\nin declaration()
\n
      if (match(FUN)) return function("function");\n
      if (match(VAR)) return varDeclaration();\n
\n
lox/Parser.java, in declaration()
\n\n

Like other statements, a function is recognized by the leading keyword. When we\nencounter fun, we call function. That corresponds to the function grammar\nrule since we already matched and consumed the fun keyword. We’ll build the\nmethod up a piece at a time, starting with this:

\n
lox/Parser.java
\nadd after expressionStatement()
\n
  private Stmt.Function function(String kind) {\n    Token name = consume(IDENTIFIER, "Expect " + kind + " name.");\n  }\n
\n
lox/Parser.java, add after expressionStatement()
\n\n

Right now, it only consumes the identifier token for the function’s name. You\nmight be wondering about that funny little kind parameter. Just like we reuse\nthe grammar rule, we’ll reuse the function() method later to parse methods\ninside classes. When we do that, we’ll pass in “method” for kind so that the\nerror messages are specific to the kind of declaration being parsed.

\n

Next, we parse the parameter list and the pair of parentheses wrapped around it.

\n
    Token name = consume(IDENTIFIER, "Expect " + kind + " name.");\n
lox/Parser.java
\nin function()
\n
    consume(LEFT_PAREN, "Expect '(' after " + kind + " name.");\n    List<Token> parameters = new ArrayList<>();\n    if (!check(RIGHT_PAREN)) {\n      do {\n        if (parameters.size() >= 255) {\n          error(peek(), "Can't have more than 255 parameters.");\n        }\n\n        parameters.add(\n            consume(IDENTIFIER, "Expect parameter name."));\n      } while (match(COMMA));\n    }\n    consume(RIGHT_PAREN, "Expect ')' after parameters.");\n
  }\n
\n
lox/Parser.java, in function()
\n\n

This is like the code for handling arguments in a call, except not split out\ninto a helper method. The outer if statement handles the zero parameter case,\nand the inner while loop parses parameters as long as we find commas to\nseparate them. The result is the list of tokens for each parameter’s name.

\n

Just like we do with arguments to function calls, we validate at parse time\nthat you don’t exceed the maximum number of parameters a function is allowed to\nhave.

\n

Finally, we parse the body and wrap it all up in a function node.

\n
    consume(RIGHT_PAREN, "Expect ')' after parameters.");\n
lox/Parser.java
\nin function()
\n
\n\n    consume(LEFT_BRACE, "Expect '{' before " + kind + " body.");\n    List<Stmt> body = block();\n    return new Stmt.Function(name, parameters, body);\n
  }\n
\n
lox/Parser.java, in function()
\n\n

Note that we consume the { at the beginning of the body here before calling\nblock(). That’s because block() assumes the brace token has already been\nmatched. Consuming it here lets us report a more precise error message if the\n{ isn’t found since we know it’s in the context of a function declaration.

\n

10 . 4Function Objects

\n

We’ve got some syntax parsed so usually we’re ready to interpret, but first we\nneed to think about how to represent a Lox function in Java. We need to keep\ntrack of the parameters so that we can bind them to argument values when the\nfunction is called. And, of course, we need to keep the code for the body of the\nfunction so that we can execute it.

\n

That’s basically what the Stmt.Function class is. Could we just use that?\nAlmost, but not quite. We also need a class that implements LoxCallable so that\nwe can call it. We don’t want the runtime phase of the interpreter to bleed into\nthe front end’s syntax classes so we don’t want Stmt.Function itself to\nimplement that. Instead, we wrap it in a new class.

\n
lox/LoxFunction.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.util.List;\n\nclass LoxFunction implements LoxCallable {\n  private final Stmt.Function declaration;\n  LoxFunction(Stmt.Function declaration) {\n    this.declaration = declaration;\n  }\n}\n
\n
lox/LoxFunction.java, create new file
\n\n

We implement the call() of LoxCallable like so:

\n
lox/LoxFunction.java
\nadd after LoxFunction()
\n
  @Override\n  public Object call(Interpreter interpreter,\n                     List<Object> arguments) {\n    Environment environment = new Environment(interpreter.globals);\n    for (int i = 0; i < declaration.params.size(); i++) {\n      environment.define(declaration.params.get(i).lexeme,\n          arguments.get(i));\n    }\n\n    interpreter.executeBlock(declaration.body, environment);\n    return null;\n  }\n
\n
lox/LoxFunction.java, add after LoxFunction()
\n\n

This handful of lines of code is one of the most fundamental, powerful pieces of\nour interpreter. As we saw in the chapter on statements and state, managing name environments is a core part\nof a language implementation. Functions are deeply tied to that.

\n\n

Parameters are core to functions, especially the fact that a function\nencapsulates its parameters—no other code outside of the function can see\nthem. This means each function gets its own environment where it stores those\nvariables.

\n

Further, this environment must be created dynamically. Each function call gets\nits own environment. Otherwise, recursion would break. If there are multiple\ncalls to the same function in play at the same time, each needs its own\nenvironment, even though they are all calls to the same function.

\n

For example, here’s a convoluted way to count to three:

\n
fun count(n) {\n  if (n > 1) count(n - 1);\n  print n;\n}\n\ncount(3);\n
\n

Imagine we pause the interpreter right at the point where it’s about to print 1\nin the innermost nested call. The outer calls to print 2 and 3 haven’t printed\ntheir values yet, so there must be environments somewhere in memory that still\nstore the fact that n is bound to 3 in one context, 2 in another, and 1 in the\ninnermost, like:

\"A\n

That’s why we create a new environment at each call, not at the function\ndeclaration. The call() method we saw earlier does that. At the beginning of\nthe call, it creates a new environment. Then it walks the parameter and argument\nlists in lockstep. For each pair, it creates a new variable with the parameter’s\nname and binds it to the argument’s value.

\n

So, for a program like this:

\n
fun add(a, b, c) {\n  print a + b + c;\n}\n\nadd(1, 2, 3);\n
\n

At the point of the call to add(), the interpreter creates something like\nthis:

\"Binding\n

Then call() tells the interpreter to execute the body of the function in this\nnew function-local environment. Up until now, the current environment was the\nenvironment where the function was being called. Now, we teleport from there\ninside the new parameter space we’ve created for the function.

\n

This is all that’s required to pass data into the function. By using different\nenvironments when we execute the body, calls to the same function with the\nsame code can produce different results.

\n

Once the body of the function has finished executing, executeBlock() discards\nthat function-local environment and restores the previous one that was active\nback at the callsite. Finally, call() returns null, which returns nil to\nthe caller. (We’ll add return values later.)

\n

Mechanically, the code is pretty simple. Walk a couple of lists. Bind some new\nvariables. Call a method. But this is where the crystalline code of the\nfunction declaration becomes a living, breathing invocation. This is one of my\nfavorite snippets in this entire book. Feel free to take a moment to meditate on\nit if you’re so inclined.

\n

Done? OK. Note when we bind the parameters, we assume the parameter and argument\nlists have the same length. This is safe because visitCallExpr() checks the\narity before calling call(). It relies on the function reporting its arity to\ndo that.

\n
lox/LoxFunction.java
\nadd after LoxFunction()
\n
  @Override\n  public int arity() {\n    return declaration.params.size();\n  }\n
\n
lox/LoxFunction.java, add after LoxFunction()
\n\n

That’s most of our object representation. While we’re in here, we may as well\nimplement toString().

\n
lox/LoxFunction.java
\nadd after LoxFunction()
\n
  @Override\n  public String toString() {\n    return "<fn " + declaration.name.lexeme + ">";\n  }\n
\n
lox/LoxFunction.java, add after LoxFunction()
\n\n

This gives nicer output if a user decides to print a function value.

\n
fun add(a, b) {\n  print a + b;\n}\n\nprint add; // "<fn add>".\n
\n

10 . 4 . 1Interpreting function declarations

\n

We’ll come back and refine LoxFunction soon, but that’s enough to get started.\nNow we can visit a function declaration.

\n
lox/Interpreter.java
\nadd after visitExpressionStmt()
\n
  @Override\n  public Void visitFunctionStmt(Stmt.Function stmt) {\n    LoxFunction function = new LoxFunction(stmt);\n    environment.define(stmt.name.lexeme, function);\n    return null;\n  }\n
\n
lox/Interpreter.java, add after visitExpressionStmt()
\n\n

This is similar to how we interpret other literal expressions. We take a\nfunction syntax node—a compile-time representation of the function—and\nconvert it to its runtime representation. Here, that’s a LoxFunction that wraps\nthe syntax node.

\n

Function declarations are different from other literal nodes in that the\ndeclaration also binds the resulting object to a new variable. So, after\ncreating the LoxFunction, we create a new binding in the current environment and\nstore a reference to it there.

\n

With that, we can define and call our own functions all within Lox. Give it a\ntry:

\n
fun sayHi(first, last) {\n  print "Hi, " + first + " " + last + "!";\n}\n\nsayHi("Dear", "Reader");\n
\n

I don’t know about you, but that looks like an honest-to-God programming\nlanguage to me.

\n

10 . 5Return Statements

\n

We can get data into functions by passing parameters, but we’ve got no way to\nget results back out. If Lox were an\nexpression-oriented language like Ruby or Scheme, the body would be an\nexpression whose value is implicitly the function’s result. But in Lox, the body\nof a function is a list of statements which don’t produce values, so we need\ndedicated syntax for emitting a result. In other words, return statements. I’m\nsure you can guess the grammar already.

\n\n
statement      → exprStmt\n               | forStmt\n               | ifStmt\n               | printStmt\n               | returnStmt\n               | whileStmt\n               | block ;\n\nreturnStmt     → "return" expression? ";" ;\n
\n

We’ve got one more—the final, in fact—production under the venerable\nstatement rule. A return statement is the return keyword followed by an\noptional expression and terminated with a semicolon.

\n

The return value is optional to support exiting early from a function that\ndoesn’t return a useful value. In statically typed languages, “void” functions\ndon’t return a value and non-void ones do. Since Lox is dynamically typed, there\nare no true void functions. The compiler has no way of preventing you from\ntaking the result value of a call to a function that doesn’t contain a return\nstatement.

\n
fun procedure() {\n  print "don't return anything";\n}\n\nvar result = procedure();\nprint result; // ?\n
\n

This means every Lox function must return something, even if it contains no\nreturn statements at all. We use nil for this, which is why LoxFunction’s\nimplementation of call() returns null at the end. In that same vein, if you\nomit the value in a return statement, we simply treat it as equivalent to:

\n
return nil;\n
\n

Over in our AST generator, we add a new node.

\n
      "Print      : Expr expression",\n
tool/GenerateAst.java
\nin main()
\n
      "Return     : Token keyword, Expr value",\n
      "Var        : Token name, Expr initializer",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

It keeps the return keyword token so we can use its location for error\nreporting, and the value being returned, if any. We parse it like other\nstatements, first by recognizing the initial keyword.

\n
    if (match(PRINT)) return printStatement();\n
lox/Parser.java
\nin statement()
\n
    if (match(RETURN)) return returnStatement();\n
    if (match(WHILE)) return whileStatement();\n
\n
lox/Parser.java, in statement()
\n\n

That branches out to:

\n
lox/Parser.java
\nadd after printStatement()
\n
  private Stmt returnStatement() {\n    Token keyword = previous();\n    Expr value = null;\n    if (!check(SEMICOLON)) {\n      value = expression();\n    }\n\n    consume(SEMICOLON, "Expect ';' after return value.");\n    return new Stmt.Return(keyword, value);\n  }\n
\n
lox/Parser.java, add after printStatement()
\n\n

After snagging the previously consumed return keyword, we look for a value\nexpression. Since many different tokens can potentially start an expression,\nit’s hard to tell if a return value is present. Instead, we check if it’s\nabsent. Since a semicolon can’t begin an expression, if the next token is\nthat, we know there must not be a value.

\n

10 . 5 . 1Returning from calls

\n

Interpreting a return statement is tricky. You can return from anywhere within\nthe body of a function, even deeply nested inside other statements. When the\nreturn is executed, the interpreter needs to jump all the way out of whatever\ncontext it’s currently in and cause the function call to complete, like some\nkind of jacked up control flow construct.

\n

For example, say we’re running this program and we’re about to execute the\nreturn statement:

\n
fun count(n) {\n  while (n < 100) {\n    if (n == 3) return n; // <--\n    print n;\n    n = n + 1;\n  }\n}\n\ncount(1);\n
\n

The Java call stack currently looks roughly like this:

\n
Interpreter.visitReturnStmt()\nInterpreter.visitIfStmt()\nInterpreter.executeBlock()\nInterpreter.visitBlockStmt()\nInterpreter.visitWhileStmt()\nInterpreter.executeBlock()\nLoxFunction.call()\nInterpreter.visitCallExpr()\n
\n

We need to get from the top of the stack all the way back to call(). I don’t\nknow about you, but to me that sounds like exceptions. When we execute a\nreturn statement, we’ll use an exception to unwind the interpreter past the\nvisit methods of all of the containing statements back to the code that began\nexecuting the body.

\n

The visit method for our new AST node looks like this:

\n
lox/Interpreter.java
\nadd after visitPrintStmt()
\n
  @Override\n  public Void visitReturnStmt(Stmt.Return stmt) {\n    Object value = null;\n    if (stmt.value != null) value = evaluate(stmt.value);\n\n    throw new Return(value);\n  }\n
\n
lox/Interpreter.java, add after visitPrintStmt()
\n\n

If we have a return value, we evaluate it, otherwise, we use nil. Then we take\nthat value and wrap it in a custom exception class and throw it.

\n
lox/Return.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nclass Return extends RuntimeException {\n  final Object value;\n\n  Return(Object value) {\n    super(null, null, false, false);\n    this.value = value;\n  }\n}\n
\n
lox/Return.java, create new file
\n\n

This class wraps the return value with the accoutrements Java requires for a\nruntime exception class. The weird super constructor call with those null and\nfalse arguments disables some JVM machinery that we don’t need. Since we’re\nusing our exception class for control flow and not\nactual error handling, we don’t need overhead like stack traces.

\n\n

We want this to unwind all the way to where the function call began, the\ncall() method in LoxFunction.

\n
          arguments.get(i));\n    }\n\n
lox/LoxFunction.java
\nin call()
\nreplace 1 line
\n
    try {\n      interpreter.executeBlock(declaration.body, environment);\n    } catch (Return returnValue) {\n      return returnValue.value;\n    }\n
    return null;\n
\n
lox/LoxFunction.java, in call(), replace 1 line
\n\n

We wrap the call to executeBlock() in a try-catch block. When it catches a\nreturn exception, it pulls out the value and makes that the return value from\ncall(). If it never catches one of these exceptions, it means the function\nreached the end of its body without hitting a return statement. In that case,\nit implicitly returns nil.

\n

Let’s try it out. We finally have enough power to support this classic\nexample—a recursive function to calculate Fibonacci numbers:

\n

\n
fun fib(n) {\n  if (n <= 1) return n;\n  return fib(n - 2) + fib(n - 1);\n}\n\nfor (var i = 0; i < 20; i = i + 1) {\n  print fib(i);\n}\n
\n

This tiny program exercises almost every language feature we have spent the past\nseveral chapters implementing—expressions, arithmetic, branching, looping,\nvariables, functions, function calls, parameter binding, and returns.

\n\n

10 . 6Local Functions and Closures

\n

Our functions are pretty full featured, but there is one hole to patch. In fact,\nit’s a big enough gap that we’ll spend most of the next chapter sealing it\nup, but we can get started here.

\n

LoxFunction’s implementation of call() creates a new environment where it\nbinds the function’s parameters. When I showed you that code, I glossed over one\nimportant point: What is the parent of that environment?

\n

Right now, it is always globals, the top-level global environment. That way,\nif an identifier isn’t defined inside the function body itself, the interpreter\ncan look outside the function in the global scope to find it. In the Fibonacci\nexample, that’s how the interpreter is able to look up the recursive call to\nfib inside the function’s own body—fib is a global variable.

\n

But recall that in Lox, function declarations are allowed anywhere a name can\nbe bound. That includes the top level of a Lox script, but also the inside of\nblocks or other functions. Lox supports local functions that are defined\ninside another function, or nested inside a block.

\n

Consider this classic example:

\n
fun makeCounter() {\n  var i = 0;\n  fun count() {\n    i = i + 1;\n    print i;\n  }\n\n  return count;\n}\n\nvar counter = makeCounter();\ncounter(); // "1".\ncounter(); // "2".\n
\n

Here, count() uses i, which is declared outside of itself in the containing\nfunction makeCounter(). makeCounter() returns a reference to the count()\nfunction and then its own body finishes executing completely.

\n

Meanwhile, the top-level code invokes the returned count() function. That\nexecutes the body of count(), which assigns to and reads i, even though the\nfunction where i was defined has already exited.

\n

If you’ve never encountered a language with nested functions before, this might\nseem crazy, but users do expect it to work. Alas, if you run it now, you get an\nundefined variable error in the call to counter() when the body of count()\ntries to look up i. That’s because the environment chain in effect looks like\nthis:

\"The\n

When we call count() (through the reference to it stored in counter), we\ncreate a new empty environment for the function body. The parent of that is the\nglobal environment. We lost the environment for makeCounter() where i is\nbound.

\n

Let’s go back in time a bit. Here’s what the environment chain looked like right\nwhen we declared count() inside the body of makeCounter():

\"The\n

So at the point where the function is declared, we can see i. But when we\nreturn from makeCounter() and exit its body, the interpreter discards that\nenvironment. Since the interpreter doesn’t keep the environment surrounding\ncount() around, it’s up to the function object itself to hang on to it.

\n

This data structure is called a closure because\nit “closes over” and holds on to the surrounding variables where the function is\ndeclared. Closures have been around since the early Lisp days, and language\nhackers have come up with all manner of ways to implement them. For jlox, we’ll\ndo the simplest thing that works. In LoxFunction, we add a field to store an\nenvironment.

\n\n
  private final Stmt.Function declaration;\n
lox/LoxFunction.java
\nin class LoxFunction
\n
  private final Environment closure;\n\n
  LoxFunction(Stmt.Function declaration) {\n
\n
lox/LoxFunction.java, in class LoxFunction
\n\n

We initialize that in the constructor.

\n
lox/LoxFunction.java
\nconstructor LoxFunction()
\nreplace 1 line
\n
  LoxFunction(Stmt.Function declaration, Environment closure) {\n    this.closure = closure;\n
    this.declaration = declaration;\n
\n
lox/LoxFunction.java, constructor LoxFunction(), replace 1 line
\n\n

When we create a LoxFunction, we capture the current environment.

\n
  public Void visitFunctionStmt(Stmt.Function stmt) {\n
lox/Interpreter.java
\nin visitFunctionStmt()
\nreplace 1 line
\n
    LoxFunction function = new LoxFunction(stmt, environment);\n
    environment.define(stmt.name.lexeme, function);\n
\n
lox/Interpreter.java, in visitFunctionStmt(), replace 1 line
\n\n

This is the environment that is active when the function is declared not when\nit’s called, which is what we want. It represents the lexical scope\nsurrounding the function declaration. Finally, when we call the function, we use\nthat environment as the call’s parent instead of going straight to globals.

\n
                     List<Object> arguments) {\n
lox/LoxFunction.java
\nin call()
\nreplace 1 line
\n
    Environment environment = new Environment(closure);\n
    for (int i = 0; i < declaration.params.size(); i++) {\n
\n
lox/LoxFunction.java, in call(), replace 1 line
\n\n

This creates an environment chain that goes from the function’s body out through\nthe environments where the function is declared, all the way out to the global\nscope. The runtime environment chain matches the textual nesting of the source\ncode like we want. The end result when we call that function looks like this:

\"The\n

Now, as you can see, the interpreter can still find i when it needs to because\nit’s in the middle of the environment chain. Try running that makeCounter()\nexample now. It works!

\n

Functions let us abstract over, reuse, and compose code. Lox is much more\npowerful than the rudimentary arithmetic calculator it used to be. Alas, in our\nrush to cram closures in, we have let a tiny bit of dynamic scoping leak into\nthe interpreter. In the next chapter, we will explore deeper into lexical\nscope and close that hole.

\n
\n

Challenges

\n
    \n
  1. \n

    Our interpreter carefully checks that the number of arguments passed to a\nfunction matches the number of parameters it expects. Since this check is\ndone at runtime on every call, it has a performance cost. Smalltalk\nimplementations don’t have that problem. Why not?

    \n
    2. \n
  2. \n

    Lox’s function declaration syntax performs two independent operations. It\ncreates a function and also binds it to a name. This improves usability for\nthe common case where you do want to associate a name with the function.\nBut in functional-styled code, you often want to create a function to\nimmediately pass it to some other function or return it. In that case, it\ndoesn’t need a name.

    \n

    Languages that encourage a functional style usually support anonymous\nfunctions or lambdas—an expression syntax that creates a function\nwithout binding it to a name. Add anonymous function syntax to Lox so that\nthis works:

    \n
    fun thrice(fn) {\n  for (var i = 1; i <= 3; i = i + 1) {\n    fn(i);\n  }\n}\n\nthrice(fun (a) {\n  print a;\n});\n// "1".\n// "2".\n// "3".\n
    \n

    How do you handle the tricky case of an anonymous function expression\noccurring in an expression statement:

    \n
    fun () {};\n
    \n
    3. \n
  3. \n

    Is this program valid?

    \n
    fun scope(a) {\n  var a = "local";\n}\n
    \n

    In other words, are a function’s parameters in the same scope as its local\nvariables, or in an outer scope? What does Lox do? What about other\nlanguages you are familiar with? What do you think a language should do?

    \n
    4. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/garbage-collection.html", "content": "\n\n\n\nGarbage Collection · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
26
\n

Garbage Collection

\n\n
\n

I wanna, I wanna,
\nI wanna, I wanna,
\nI wanna be trash.

\n

The Whip, “Trash”

\n
\n

We say Lox is a “high-level” language because it frees programmers from worrying\nabout details irrelevant to the problem they’re solving. The user becomes an\nexecutive, giving the machine abstract goals and letting the lowly computer\nfigure out how to get there.

\n

Dynamic memory allocation is a perfect candidate for automation. It’s necessary\nfor a working program, tedious to do by hand, and yet still error-prone. The\ninevitable mistakes can be catastrophic, leading to crashes, memory corruption,\nor security violations. It’s the kind of risky-yet-boring work that machines\nexcel at over humans.

\n

This is why Lox is a managed language, which means that the language\nimplementation manages memory allocation and freeing on the user’s behalf. When\na user performs an operation that requires some dynamic memory, the VM\nautomatically allocates it. The programmer never worries about deallocating\nanything. The machine ensures any memory the program is using sticks around as\nlong as needed.

\n

Lox provides the illusion that the computer has an infinite amount of memory.\nUsers can allocate and allocate and allocate and never once think about where\nall these bytes are coming from. Of course, computers do not yet have infinite\nmemory. So the way managed languages maintain this illusion is by going behind\nthe programmer’s back and reclaiming memory that the program no longer needs.\nThe component that does this is called a garbage collector.

\n\n

26 . 1Reachability

\n

This raises a surprisingly difficult question: how does a VM tell what memory is\nnot needed? Memory is only needed if it is read in the future, but short of\nhaving a time machine, how can an implementation tell what code the program\nwill execute and which data it will use? Spoiler alert: VMs cannot travel\ninto the future. Instead, the language makes a conservative approximation: it considers a piece of\nmemory to still be in use if it could possibly be read in the future.

\n\n

That sounds too conservative. Couldn’t any bit of memory potentially be\nread? Actually, no, at least not in a memory-safe language like Lox. Here’s an\nexample:

\n
var a = "first value";\na = "updated";\n// GC here.\nprint a;\n
\n

Say we run the GC after the assignment has completed on the second line. The\nstring “first value” is still sitting in memory, but there is no way for the\nuser’s program to ever get to it. Once a got reassigned, the program lost any\nreference to that string. We can safely free it. A value is reachable if\nthere is some way for a user program to reference it. Otherwise, like the string\n“first value” here, it is unreachable.

\n

Many values can be directly accessed by the VM. Take a look at:

\n
var global = "string";\n{\n  var local = "another";\n  print global + local;\n}\n
\n

Pause the program right after the two strings have been concatenated but before\nthe print statement has executed. The VM can reach \"string\" by looking\nthrough the global variable table and finding the entry for global. It can\nfind \"another\" by walking the value stack and hitting the slot for the local\nvariable local. It can even find the concatenated string \"stringanother\"\nsince that temporary value is also sitting on the VM’s stack at the point when\nwe paused our program.

\n

All of these values are called roots. A root is any object that the VM can\nreach directly without going through a reference in some other object. Most\nroots are global variables or on the stack, but as we’ll see, there are a couple\nof other places the VM stores references to objects that it can find.

\n

Other values can be found by going through a reference inside another value.\nFields on instances of classes are the most obvious\ncase, but we don’t have those yet. Even without those, our VM still has indirect\nreferences. Consider:

\n\n
fun makeClosure() {\n  var a = "data";\n\n  fun f() { print a; }\n  return f;\n}\n\n{\n  var closure = makeClosure();\n  // GC here.\n  closure();\n}\n
\n

Say we pause the program on the marked line and run the garbage collector. When\nthe collector is done and the program resumes, it will call the closure, which\nwill in turn print \"data\". So the collector needs to not free that string.\nBut here’s what the stack looks like when we pause the program:

\"The\n

The \"data\" string is nowhere on it. It has already been hoisted off the stack\nand moved into the closed upvalue that the closure uses. The closure itself is\non the stack. But to get to the string, we need to trace through the closure and\nits upvalue array. Since it is possible for the user’s program to do that, all\nof these indirectly accessible objects are also considered reachable.

\"All\n

This gives us an inductive definition of reachability:

\n
    \n
  • \n

    All roots are reachable.

    \n
    • \n
  • \n

    Any object referred to from a reachable object is itself reachable.

    \n
    • \n
\n

These are the values that are still “live” and need to stay in memory. Any value\nthat doesn’t meet this definition is fair game for the collector to reap.\nThat recursive pair of rules hints at a recursive algorithm we can use to free\nup unneeded memory:

\n
    \n
  1. \n

    Starting with the roots, traverse through object references to find the\nfull set of reachable objects.

    \n
    2. \n
  2. \n

    Free all objects not in that set.

    \n
    3. \n
\n

Many different garbage collection algorithms are in\nuse today, but they all roughly follow that same structure. Some may interleave\nthe steps or mix them, but the two fundamental operations are there. They mostly\ndiffer in how they perform each step.

\n\n

26 . 2Mark-Sweep Garbage Collection

\n

The first managed language was Lisp, the second “high-level” language to be\ninvented, right after Fortran. John McCarthy considered using manual memory\nmanagement or reference counting, but eventually settled on (and coined) garbage\ncollection—once the program was out of memory, it would go back and find\nunused storage it could reclaim.

\n\n

He designed the very first, simplest garbage collection algorithm, called\nmark-and-sweep or just mark-sweep. Its description fits in three short\nparagraphs in the initial paper on Lisp. Despite its age and simplicity, the\nsame fundamental algorithm underlies many modern memory managers. Some corners\nof CS seem to be timeless.

\n

As the name implies, mark-sweep works in two phases:

\n
    \n
  • \n

    Marking: We start with the roots and traverse or trace through all of the objects those roots refer to.\nThis is a classic graph traversal of all of the reachable objects. Each time\nwe visit an object, we mark it in some way. (Implementations differ in how\nthey record the mark.)

    \n
    • \n
  • \n

    Sweeping: Once the mark phase completes, every reachable object\nin the heap has been marked. That means any unmarked object is unreachable and\nripe for reclamation. We go through all the unmarked objects and free each\none.

    \n
    • \n
\n

It looks something like this:

\"Starting\n\n

That’s what we’re gonna implement. Whenever we decide it’s time to reclaim some\nbytes, we’ll trace everything and mark all the reachable objects, free what\ndidn’t get marked, and then resume the user’s program.

\n

26 . 2 . 1Collecting garbage

\n

This entire chapter is about implementing this one function:

\n\n
void* reallocate(void* pointer, size_t oldSize, size_t newSize);\n
memory.h
\nadd after reallocate()
\n
void collectGarbage();\n
void freeObjects();\n
\n
memory.h, add after reallocate()
\n\n

We’ll work our way up to a full implementation starting with this empty shell:

\n
memory.c
\nadd after freeObject()
\n
void collectGarbage() {\n}\n
\n
memory.c, add after freeObject()
\n\n

The first question you might ask is, When does this function get called? It\nturns out that’s a subtle question that we’ll spend some time on later in the\nchapter. For now we’ll sidestep the issue and build ourselves a handy diagnostic\ntool in the process.

\n
#define DEBUG_TRACE_EXECUTION\n
common.h
\n
\n\n#define DEBUG_STRESS_GC\n
\n\n#define UINT8_COUNT (UINT8_MAX + 1)\n
\n
common.h
\n\n

We’ll add an optional “stress test” mode for the garbage collector. When this\nflag is defined, the GC runs as often as it possibly can. This is, obviously,\nhorrendous for performance. But it’s great for flushing out memory management\nbugs that occur only when a GC is triggered at just the right moment. If every\nmoment triggers a GC, you’re likely to find those bugs.

\n
void* reallocate(void* pointer, size_t oldSize, size_t newSize) {\n
memory.c
\nin reallocate()
\n
  if (newSize > oldSize) {\n#ifdef DEBUG_STRESS_GC\n    collectGarbage();\n#endif\n  }\n\n
  if (newSize == 0) {\n
\n
memory.c, in reallocate()
\n\n

Whenever we call reallocate() to acquire more memory, we force a collection to\nrun. The if check is because reallocate() is also called to free or shrink an\nallocation. We don’t want to trigger a GC for that—in particular because the\nGC itself will call reallocate() to free memory.

\n

Collecting right before allocation is the classic way\nto wire a GC into a VM. You’re already calling into the memory manager, so it’s\nan easy place to hook in the code. Also, allocation is the only time when you\nreally need some freed up memory so that you can reuse it. If you don’t use\nallocation to trigger a GC, you have to make sure every possible place in code\nwhere you can loop and allocate memory also has a way to trigger the collector.\nOtherwise, the VM can get into a starved state where it needs more memory but\nnever collects any.

\n\n

26 . 2 . 2Debug logging

\n

While we’re on the subject of diagnostics, let’s put some more in. A real\nchallenge I’ve found with garbage collectors is that they are opaque. We’ve been\nrunning lots of Lox programs just fine without any GC at all so far. Once we\nadd one, how do we tell if it’s doing anything useful? Can we tell only if we\nwrite programs that plow through acres of memory? How do we debug that?

\n

An easy way to shine a light into the GC’s inner workings is with some logging.

\n
#define DEBUG_STRESS_GC\n
common.h
\n
#define DEBUG_LOG_GC\n
\n\n#define UINT8_COUNT (UINT8_MAX + 1)\n
\n
common.h
\n\n

When this is enabled, clox prints information to the console when it does\nsomething with dynamic memory.

\n

We need a couple of includes.

\n
#include "vm.h"\n
memory.c
\n
\n\n#ifdef DEBUG_LOG_GC\n#include <stdio.h>\n#include "debug.h"\n#endif\n
\n\nvoid* reallocate(void* pointer, size_t oldSize, size_t newSize) {\n
\n
memory.c
\n\n

We don’t have a collector yet, but we can start putting in some of the logging\nnow. We’ll want to know when a collection run starts.

\n
void collectGarbage() {\n
memory.c
\nin collectGarbage()
\n
#ifdef DEBUG_LOG_GC\n  printf("-- gc begin\\n");\n#endif\n
}\n
\n
memory.c, in collectGarbage()
\n\n

Eventually we will log some other operations during the collection, so we’ll\nalso want to know when the show’s over.

\n
  printf("-- gc begin\\n");\n#endif\n
memory.c
\nin collectGarbage()
\n
\n\n#ifdef DEBUG_LOG_GC\n  printf("-- gc end\\n");\n#endif\n
}\n
\n
memory.c, in collectGarbage()
\n\n

We don’t have any code for the collector yet, but we do have functions for\nallocating and freeing, so we can instrument those now.

\n
  vm.objects = object;\n
object.c
\nin allocateObject()
\n
\n\n#ifdef DEBUG_LOG_GC\n  printf("%p allocate %zu for %d\\n", (void*)object, size, type);\n#endif\n\n
  return object;\n
\n
object.c, in allocateObject()
\n\n

And at the end of an object’s lifespan:

\n
static void freeObject(Obj* object) {\n
memory.c
\nin freeObject()
\n
#ifdef DEBUG_LOG_GC\n  printf("%p free type %d\\n", (void*)object, object->type);\n#endif\n\n
  switch (object->type) {\n
\n
memory.c, in freeObject()
\n\n

With these two flags, we should be able to see that we’re making progress as we\nwork through the rest of the chapter.

\n

26 . 3Marking the Roots

\n

Objects are scattered across the heap like stars in the inky night sky. A\nreference from one object to another forms a connection, and these\nconstellations are the graph that the mark phase traverses. Marking begins at\nthe roots.

\n
#ifdef DEBUG_LOG_GC\n  printf("-- gc begin\\n");\n#endif\n
memory.c
\nin collectGarbage()
\n
\n\n  markRoots();\n
\n\n#ifdef DEBUG_LOG_GC\n
\n
memory.c, in collectGarbage()
\n\n

Most roots are local variables or temporaries sitting right in the VM’s stack,\nso we start by walking that.

\n
memory.c
\nadd after freeObject()
\n
static void markRoots() {\n  for (Value* slot = vm.stack; slot < vm.stackTop; slot++) {\n    markValue(*slot);\n  }\n}\n
\n
memory.c, add after freeObject()
\n\n

To mark a Lox value, we use this new function:

\n
void* reallocate(void* pointer, size_t oldSize, size_t newSize);\n
memory.h
\nadd after reallocate()
\n
void markValue(Value value);\n
void collectGarbage();\n
\n
memory.h, add after reallocate()
\n\n

Its implementation is here:

\n
memory.c
\nadd after reallocate()
\n
void markValue(Value value) {\n  if (IS_OBJ(value)) markObject(AS_OBJ(value));\n}\n
\n
memory.c, add after reallocate()
\n\n

Some Lox values—numbers, Booleans, and nil—are stored directly inline in\nValue and require no heap allocation. The garbage collector doesn’t need to\nworry about them at all, so the first thing we do is ensure that the value is an\nactual heap object. If so, the real work happens in this function:

\n
void* reallocate(void* pointer, size_t oldSize, size_t newSize);\n
memory.h
\nadd after reallocate()
\n
void markObject(Obj* object);\n
void markValue(Value value);\n
\n
memory.h, add after reallocate()
\n\n

Which is defined here:

\n
memory.c
\nadd after reallocate()
\n
void markObject(Obj* object) {\n  if (object == NULL) return;\n  object->isMarked = true;\n}\n
\n
memory.c, add after reallocate()
\n\n

The NULL check is unnecessary when called from markValue(). A Lox Value that\nis some kind of Obj type will always have a valid pointer. But later we will\ncall this function directly from other code, and in some of those places, the\nobject being pointed to is optional.

\n

Assuming we do have a valid object, we mark it by setting a flag. That new field\nlives in the Obj header struct all objects share.

\n
  ObjType type;\n
object.h
\nin struct Obj
\n
  bool isMarked;\n
  struct Obj* next;\n
\n
object.h, in struct Obj
\n\n

Every new object begins life unmarked because we haven’t yet determined if it is\nreachable or not.

\n
  object->type = type;\n
object.c
\nin allocateObject()
\n
  object->isMarked = false;\n
\n\n  object->next = vm.objects;\n
\n
object.c, in allocateObject()
\n\n

Before we go any farther, let’s add some logging to markObject().

\n
void markObject(Obj* object) {\n  if (object == NULL) return;\n
memory.c
\nin markObject()
\n
#ifdef DEBUG_LOG_GC\n  printf("%p mark ", (void*)object);\n  printValue(OBJ_VAL(object));\n  printf("\\n");\n#endif\n\n
  object->isMarked = true;\n
\n
memory.c, in markObject()
\n\n

This way we can see what the mark phase is doing. Marking the stack takes care\nof local variables and temporaries. The other main source of roots are the\nglobal variables.

\n
    markValue(*slot);\n  }\n
memory.c
\nin markRoots()
\n
\n\n  markTable(&vm.globals);\n
}\n
\n
memory.c, in markRoots()
\n\n

Those live in a hash table owned by the VM, so we’ll declare another helper\nfunction for marking all of the objects in a table.

\n
ObjString* tableFindString(Table* table, const char* chars,\n                           int length, uint32_t hash);\n
table.h
\nadd after tableFindString()
\n
void markTable(Table* table);\n
\n\n#endif\n
\n
table.h, add after tableFindString()
\n\n

We implement that in the “table” module here:

\n
table.c
\nadd after tableFindString()
\n
void markTable(Table* table) {\n  for (int i = 0; i < table->capacity; i++) {\n    Entry* entry = &table->entries[i];\n    markObject((Obj*)entry->key);\n    markValue(entry->value);\n  }\n}\n
\n
table.c, add after tableFindString()
\n\n

Pretty straightforward. We walk the entry array. For each one, we mark its\nvalue. We also mark the key strings for each entry since the GC manages those\nstrings too.

\n

26 . 3 . 1Less obvious roots

\n

Those cover the roots that we typically think of—the values that are\nobviously reachable because they’re stored in variables the user’s program can\nsee. But the VM has a few of its own hidey-holes where it squirrels away\nreferences to values that it directly accesses.

\n

Most function call state lives in the value stack, but the VM maintains a\nseparate stack of CallFrames. Each CallFrame contains a pointer to the closure\nbeing called. The VM uses those pointers to access constants and upvalues, so\nthose closures need to be kept around too.

\n
  }\n
memory.c
\nin markRoots()
\n
\n\n  for (int i = 0; i < vm.frameCount; i++) {\n    markObject((Obj*)vm.frames[i].closure);\n  }\n
\n\n  markTable(&vm.globals);\n
\n
memory.c, in markRoots()
\n\n

Speaking of upvalues, the open upvalue list is another set of values that the\nVM can directly reach.

\n
  for (int i = 0; i < vm.frameCount; i++) {\n    markObject((Obj*)vm.frames[i].closure);\n  }\n
memory.c
\nin markRoots()
\n
\n\n  for (ObjUpvalue* upvalue = vm.openUpvalues;\n       upvalue != NULL;\n       upvalue = upvalue->next) {\n    markObject((Obj*)upvalue);\n  }\n
\n\n  markTable(&vm.globals);\n
\n
memory.c, in markRoots()
\n\n

Remember also that a collection can begin during any allocation. Those\nallocations don’t just happen while the user’s program is running. The compiler\nitself periodically grabs memory from the heap for literals and the constant\ntable. If the GC runs while we’re in the middle of compiling, then any values\nthe compiler directly accesses need to be treated as roots too.

\n

To keep the compiler module cleanly separated from the rest of the VM, we’ll do\nthat in a separate function.

\n
  markTable(&vm.globals);\n
memory.c
\nin markRoots()
\n
  markCompilerRoots();\n
}\n
\n
memory.c, in markRoots()
\n\n

It’s declared here:

\n
ObjFunction* compile(const char* source);\n
compiler.h
\nadd after compile()
\n
void markCompilerRoots();\n
\n\n#endif\n
\n
compiler.h, add after compile()
\n\n

Which means the “memory” module needs an include.

\n
#include <stdlib.h>\n\n
memory.c
\n
#include "compiler.h"\n
#include "memory.h"\n
\n
memory.c
\n\n

And the definition is over in the “compiler” module.

\n
compiler.c
\nadd after compile()
\n
void markCompilerRoots() {\n  Compiler* compiler = current;\n  while (compiler != NULL) {\n    markObject((Obj*)compiler->function);\n    compiler = compiler->enclosing;\n  }\n}\n
\n
compiler.c, add after compile()
\n\n

Fortunately, the compiler doesn’t have too many values that it hangs on to. The\nonly object it uses is the ObjFunction it is compiling into. Since function\ndeclarations can nest, the compiler has a linked list of those and we walk the\nwhole list.

\n

Since the “compiler” module is calling markObject(), it also needs an include.

\n
#include "compiler.h"\n
compiler.c
\n
#include "memory.h"\n
#include "scanner.h"\n
\n
compiler.c
\n\n

Those are all the roots. After running this, every object that the VM—runtime\nand compiler—can get to without going through some other object has its\nmark bit set.

\n

26 . 4Tracing Object References

\n

The next step in the marking process is tracing through the graph of references\nbetween objects to find the indirectly reachable values. We don’t have instances\nwith fields yet, so there aren’t many objects that contain references, but we do\nhave some. In particular, ObjClosure has the list of\nObjUpvalues it closes over as well as a reference to the raw ObjFunction that it\nwraps. ObjFunction, in turn, has a constant table containing references to all\nof the literals created in the function’s body. This is enough to build a fairly\ncomplex web of objects for our collector to crawl through.

\n\n

Now it’s time to implement that traversal. We can go breadth-first, depth-first,\nor in some other order. Since we just need to find the set of all reachable\nobjects, the order we visit them mostly doesn’t matter.

\n\n

26 . 4 . 1The tricolor abstraction

\n

As the collector wanders through the graph of objects, we need to make sure it\ndoesn’t lose track of where it is or get stuck going in circles. This is\nparticularly a concern for advanced implementations like incremental GCs that\ninterleave marking with running pieces of the user’s program. The collector\nneeds to be able to pause and then pick up where it left off later.

\n

To help us soft-brained humans reason about this complex process, VM hackers\ncame up with a metaphor called the tricolor\nabstraction. Each object has a conceptual “color” that tracks what state the\nobject is in, and what work is left to do.

\n\n
    \n
  • \n

    \"A White: At the beginning of a garbage collection, every\nobject is white. This color means we have not reached or processed the\nobject at all.

    \n
    • \n
  • \n

    \"A Gray: During marking, when we first reach an object, we\ndarken it gray. This color means we know the object itself is reachable and\nshould not be collected. But we have not yet traced through it to see what\nother objects it references. In graph algorithm terms, this is the\nworklist—the set of objects we know about but haven’t processed yet.

    \n
    • \n
  • \n

    \"A Black: When\nwe take a gray object and mark all of the objects it references, we then\nturn the gray object black. This color means the mark phase is done\nprocessing that object.

    \n
    • \n
\n

In terms of that abstraction, the marking process now looks like this:

\n
    \n
  1. \n

    Start off with all objects white.

    \n
    2. \n
  2. \n

    Find all the roots and mark them gray.

    \n
    3. \n
  3. \n

    Repeat as long as there are still gray objects:

    \n
      \n
    1. \n

      Pick a gray object. Turn any white objects that the object mentions to\ngray.

      \n
      2. \n
    2. \n

      Mark the original gray object black.

      \n
      3. \n
    \n
    4. \n
\n

I find it helps to visualize this. You have a web of objects with references\nbetween them. Initially, they are all little white dots. Off to the side are\nsome incoming edges from the VM that point to the roots. Those roots turn gray.\nThen each gray object’s siblings turn gray while the object itself turns black.\nThe full effect is a gray wavefront that passes through the graph, leaving a\nfield of reachable black objects behind it. Unreachable objects are not touched\nby the wavefront and stay white.

\"A\n

At the end, you’re left with a sea of reached,\nblack objects sprinkled with islands of white objects that can be swept up and\nfreed. Once the unreachable objects are freed, the remaining objects—all\nblack—are reset to white for the next garbage collection cycle.

\n\n

26 . 4 . 2A worklist for gray objects

\n

In our implementation we have already marked the roots. They’re all gray. The\nnext step is to start picking them and traversing their references. But we don’t\nhave any easy way to find them. We set a field on the object, but that’s it. We\ndon’t want to have to traverse the entire object list looking for objects with\nthat field set.

\n

Instead, we’ll create a separate worklist to keep track of all of the gray\nobjects. When an object turns gray, in addition to setting the mark field we’ll\nalso add it to the worklist.

\n
  object->isMarked = true;\n
memory.c
\nin markObject()
\n
\n\n  if (vm.grayCapacity < vm.grayCount + 1) {\n    vm.grayCapacity = GROW_CAPACITY(vm.grayCapacity);\n    vm.grayStack = (Obj**)realloc(vm.grayStack,\n                                  sizeof(Obj*) * vm.grayCapacity);\n  }\n\n  vm.grayStack[vm.grayCount++] = object;\n
}\n
\n
memory.c, in markObject()
\n\n

We could use any kind of data structure that lets us put items in and take them\nout easily. I picked a stack because that’s the simplest to implement with a\ndynamic array in C. It works mostly like other dynamic arrays we’ve built in\nLox, except, note that it calls the system realloc() function and not our\nown reallocate() wrapper. The memory for the gray stack itself is not\nmanaged by the garbage collector. We don’t want growing the gray stack during a\nGC to cause the GC to recursively start a new GC. That could tear a hole in the\nspace-time continuum.

\n

We’ll manage its memory ourselves, explicitly. The VM owns the gray stack.

\n
  Obj* objects;\n
vm.h
\nin struct VM
\n
  int grayCount;\n  int grayCapacity;\n  Obj** grayStack;\n
} VM;\n
\n
vm.h, in struct VM
\n\n

It starts out empty.

\n
  vm.objects = NULL;\n
vm.c
\nin initVM()
\n
\n\n  vm.grayCount = 0;\n  vm.grayCapacity = 0;\n  vm.grayStack = NULL;\n
\n\n  initTable(&vm.globals);\n
\n
vm.c, in initVM()
\n\n

And we need to free it when the VM shuts down.

\n
    object = next;\n  }\n
memory.c
\nin freeObjects()
\n
\n\n  free(vm.grayStack);\n
}\n
\n
memory.c, in freeObjects()
\n\n

We take full responsibility for this array. That\nincludes allocation failure. If we can’t create or grow the gray stack, then we\ncan’t finish the garbage collection. This is bad news for the VM, but\nfortunately rare since the gray stack tends to be pretty small. It would be nice\nto do something more graceful, but to keep the code in this book simple, we just\nabort.

\n\n
    vm.grayStack = (Obj**)realloc(vm.grayStack,\n                                  sizeof(Obj*) * vm.grayCapacity);\n
memory.c
\nin markObject()
\n
\n\n    if (vm.grayStack == NULL) exit(1);\n
  }\n
\n
memory.c, in markObject()
\n\n

26 . 4 . 3Processing gray objects

\n

OK, now when we’re done marking the roots, we have both set a bunch of fields\nand filled our work list with objects to chew through. It’s time for the next\nphase.

\n
  markRoots();\n
memory.c
\nin collectGarbage()
\n
  traceReferences();\n
\n\n#ifdef DEBUG_LOG_GC\n
\n
memory.c, in collectGarbage()
\n\n

Here’s the implementation:

\n
memory.c
\nadd after markRoots()
\n
static void traceReferences() {\n  while (vm.grayCount > 0) {\n    Obj* object = vm.grayStack[--vm.grayCount];\n    blackenObject(object);\n  }\n}\n
\n
memory.c, add after markRoots()
\n\n

It’s as close to that textual algorithm as you can get. Until the stack empties,\nwe keep pulling out gray objects, traversing their references, and then marking\nthem black. Traversing an object’s references may turn up new white objects that\nget marked gray and added to the stack. So this function swings back and forth\nbetween turning white objects gray and gray objects black, gradually advancing\nthe entire wavefront forward.

\n

Here’s where we traverse a single object’s references:

\n
memory.c
\nadd after markValue()
\n
static void blackenObject(Obj* object) {\n  switch (object->type) {\n    case OBJ_NATIVE:\n    case OBJ_STRING:\n      break;\n  }\n}\n
\n
memory.c, add after markValue()
\n\n

Each object kind has different fields that might\nreference other objects, so we need a specific blob of code for each type. We\nstart with the easy ones—strings and native function objects contain no\noutgoing references so there is nothing to traverse.

\n\n

Note that we don’t set any state in the traversed object itself. There is no\ndirect encoding of “black” in the object’s state. A black object is any object\nwhose isMarked field is set and that is no longer in\nthe gray stack.

\n\n

Now let’s start adding in the other object types. The simplest is upvalues.

\n
static void blackenObject(Obj* object) {\n  switch (object->type) {\n
memory.c
\nin blackenObject()
\n
    case OBJ_UPVALUE:\n      markValue(((ObjUpvalue*)object)->closed);\n      break;\n
    case OBJ_NATIVE:\n
\n
memory.c, in blackenObject()
\n\n

When an upvalue is closed, it contains a reference to the closed-over value.\nSince the value is no longer on the stack, we need to make sure we trace the\nreference to it from the upvalue.

\n

Next are functions.

\n
  switch (object->type) {\n
memory.c
\nin blackenObject()
\n
    case OBJ_FUNCTION: {\n      ObjFunction* function = (ObjFunction*)object;\n      markObject((Obj*)function->name);\n      markArray(&function->chunk.constants);\n      break;\n    }\n
    case OBJ_UPVALUE:\n
\n
memory.c, in blackenObject()
\n\n

Each function has a reference to an ObjString containing the function’s name.\nMore importantly, the function has a constant table packed full of references to\nother objects. We trace all of those using this helper:

\n
memory.c
\nadd after markValue()
\n
static void markArray(ValueArray* array) {\n  for (int i = 0; i < array->count; i++) {\n    markValue(array->values[i]);\n  }\n}\n
\n
memory.c, add after markValue()
\n\n

The last object type we have now—we’ll add more in later chapters—is\nclosures.

\n
  switch (object->type) {\n
memory.c
\nin blackenObject()
\n
    case OBJ_CLOSURE: {\n      ObjClosure* closure = (ObjClosure*)object;\n      markObject((Obj*)closure->function);\n      for (int i = 0; i < closure->upvalueCount; i++) {\n        markObject((Obj*)closure->upvalues[i]);\n      }\n      break;\n    }\n
    case OBJ_FUNCTION: {\n
\n
memory.c, in blackenObject()
\n\n

Each closure has a reference to the bare function it wraps, as well as an array\nof pointers to the upvalues it captures. We trace all of those.

\n

That’s the basic mechanism for processing a gray object, but there are two loose\nends to tie up. First, some logging.

\n
static void blackenObject(Obj* object) {\n
memory.c
\nin blackenObject()
\n
#ifdef DEBUG_LOG_GC\n  printf("%p blacken ", (void*)object);\n  printValue(OBJ_VAL(object));\n  printf("\\n");\n#endif\n\n
  switch (object->type) {\n
\n
memory.c, in blackenObject()
\n\n

This way, we can watch the tracing percolate through the object graph. Speaking\nof which, note that I said graph. References between objects are directed, but\nthat doesn’t mean they’re acyclic! It’s entirely possible to have cycles of\nobjects. When that happens, we need to ensure our collector doesn’t get stuck in\nan infinite loop as it continually re-adds the same series of objects to the\ngray stack.

\n

The fix is easy.

\n
  if (object == NULL) return;\n
memory.c
\nin markObject()
\n
  if (object->isMarked) return;\n\n
#ifdef DEBUG_LOG_GC\n
\n
memory.c, in markObject()
\n\n

If the object is already marked, we don’t mark it again and thus don’t add it to\nthe gray stack. This ensures that an already-gray object is not redundantly\nadded and that a black object is not inadvertently turned back to gray. In other\nwords, it keeps the wavefront moving forward through only the white objects.

\n

26 . 5Sweeping Unused Objects

\n

When the loop in traceReferences() exits, we have processed all the objects we\ncould get our hands on. The gray stack is empty, and every object in the heap is\neither black or white. The black objects are reachable, and we want to hang on to\nthem. Anything still white never got touched by the trace and is thus garbage.\nAll that’s left is to reclaim them.

\n
  traceReferences();\n
memory.c
\nin collectGarbage()
\n
  sweep();\n
\n\n#ifdef DEBUG_LOG_GC\n
\n
memory.c, in collectGarbage()
\n\n

All of the logic lives in one function.

\n
memory.c
\nadd after traceReferences()
\n
static void sweep() {\n  Obj* previous = NULL;\n  Obj* object = vm.objects;\n  while (object != NULL) {\n    if (object->isMarked) {\n      previous = object;\n      object = object->next;\n    } else {\n      Obj* unreached = object;\n      object = object->next;\n      if (previous != NULL) {\n        previous->next = object;\n      } else {\n        vm.objects = object;\n      }\n\n      freeObject(unreached);\n    }\n  }\n}\n
\n
memory.c, add after traceReferences()
\n\n

I know that’s kind of a lot of code and pointer shenanigans, but there isn’t\nmuch to it once you work through it. The outer while loop walks the linked\nlist of every object in the heap, checking their mark bits. If an object is\nmarked (black), we leave it alone and continue past it. If it is unmarked\n(white), we unlink it from the list and free it using the freeObject()\nfunction we already wrote.

\"A\n

Most of the other code in here deals with the fact that removing a node from a\nsingly linked list is cumbersome. We have to continuously remember the previous\nnode so we can unlink its next pointer, and we have to handle the edge case\nwhere we are freeing the first node. But, otherwise, it’s pretty simple—delete every node in a linked list that doesn’t have a bit set in it.

\n

There’s one little addition:

\n
    if (object->isMarked) {\n
memory.c
\nin sweep()
\n
      object->isMarked = false;\n
      previous = object;\n
\n
memory.c, in sweep()
\n\n

After sweep() completes, the only remaining objects are the live black ones\nwith their mark bits set. That’s correct, but when the next collection cycle\nstarts, we need every object to be white. So whenever we reach a black object,\nwe go ahead and clear the bit now in anticipation of the next run.

\n

26 . 5 . 1Weak references and the string pool

\n

We are almost done collecting. There is one remaining corner of the VM that has\nsome unusual requirements around memory. Recall that when we added strings to\nclox we made the VM intern them all. That means the VM has a hash table\ncontaining a pointer to every single string in the heap. The VM uses this to\nde-duplicate strings.

\n

During the mark phase, we deliberately did not treat the VM’s string table as\na source of roots. If we had, no string would ever\nbe collected. The string table would grow and grow and never yield a single byte\nof memory back to the operating system. That would be bad.

\n\n

At the same time, if we do let the GC free strings, then the VM’s string table\nwill be left with dangling pointers to freed memory. That would be even worse.

\n

The string table is special and we need special support for it. In particular,\nit needs a special kind of reference. The table should be able to refer to a\nstring, but that link should not be considered a root when determining\nreachability. That implies that the referenced object can be freed. When that\nhappens, the dangling reference must be fixed too, sort of like a magic,\nself-clearing pointer. This particular set of semantics comes up frequently\nenough that it has a name: a weak reference.

\n

We have already implicitly implemented half of the string table’s unique\nbehavior by virtue of the fact that we don’t traverse it during marking. That\nmeans it doesn’t force strings to be reachable. The remaining piece is clearing\nout any dangling pointers for strings that are freed.

\n

To remove references to unreachable strings, we need to know which strings are\nunreachable. We don’t know that until after the mark phase has completed. But we\ncan’t wait until after the sweep phase is done because by then the objects—and their mark bits—are no longer around to check. So the right time is\nexactly between the marking and sweeping phases.

\n
  traceReferences();\n
memory.c
\nin collectGarbage()
\n
  tableRemoveWhite(&vm.strings);\n
  sweep();\n
\n
memory.c, in collectGarbage()
\n\n

The logic for removing the about-to-be-deleted strings exists in a new function\nin the “table” module.

\n
ObjString* tableFindString(Table* table, const char* chars,\n                           int length, uint32_t hash);\n
table.h
\nadd after tableFindString()
\n
\n\nvoid tableRemoveWhite(Table* table);\n
void markTable(Table* table);\n\n
\n
table.h, add after tableFindString()
\n\n

The implementation is here:

\n
table.c
\nadd after tableFindString()
\n
void tableRemoveWhite(Table* table) {\n  for (int i = 0; i < table->capacity; i++) {\n    Entry* entry = &table->entries[i];\n    if (entry->key != NULL && !entry->key->obj.isMarked) {\n      tableDelete(table, entry->key);\n    }\n  }\n}\n
\n
table.c, add after tableFindString()
\n\n

We walk every entry in the table. The string intern table uses only the key of\neach entry—it’s basically a hash set not a hash map. If the key string\nobject’s mark bit is not set, then it is a white object that is moments from\nbeing swept away. We delete it from the hash table first and thus ensure we\nwon’t see any dangling pointers.

\n

26 . 6When to Collect

\n

We have a fully functioning mark-sweep garbage collector now. When the stress\ntesting flag is enabled, it gets called all the time, and with the logging\nenabled too, we can watch it do its thing and see that it is indeed reclaiming\nmemory. But, when the stress testing flag is off, it never runs at all. It’s\ntime to decide when the collector should be invoked during normal program\nexecution.

\n

As far as I can tell, this question is poorly answered by the literature. When\ngarbage collectors were first invented, computers had a tiny, fixed amount of\nmemory. Many of the early GC papers assumed that you set aside a few thousand\nwords of memory—in other words, most of it—and invoked the collector\nwhenever you ran out. Simple.

\n

Modern machines have gigs of physical RAM, hidden behind the operating system’s\neven larger virtual memory abstraction, which is shared among a slew of other\nprograms all fighting for their chunk of memory. The operating system will let\nyour program request as much as it wants and then page in and out from the disc\nwhen physical memory gets full. You never really “run out” of memory, you just\nget slower and slower.

\n

26 . 6 . 1Latency and throughput

\n

It no longer makes sense to wait until you “have to”, to run the GC, so we need\na more subtle timing strategy. To reason about this more precisely, it’s time to\nintroduce two fundamental numbers used when measuring a memory manager’s\nperformance: throughput and latency.

\n

Every managed language pays a performance price compared to explicit,\nuser-authored deallocation. The time spent actually freeing memory is the same,\nbut the GC spends cycles figuring out which memory to free. That is time not\nspent running the user’s code and doing useful work. In our implementation,\nthat’s the entirety of the mark phase. The goal of a sophisticated garbage\ncollector is to minimize that overhead.

\n

There are two key metrics we can use to understand that cost better:

\n
    \n
  • \n

    Throughput is the total fraction of time spent running user code versus\ndoing garbage collection work. Say you run a clox program for ten seconds\nand it spends a second of that inside collectGarbage(). That means the\nthroughput is 90%—it spent 90% of the time running the program and 10%\non GC overhead.

    \n

    Throughput is the most fundamental measure because it tracks the total cost\nof collection overhead. All else being equal, you want to maximize\nthroughput. Up until this chapter, clox had no GC at all and thus 100% throughput. That’s pretty hard to beat. Of\ncourse, it came at the slight expense of potentially running out of memory\nand crashing if the user’s program ran long enough. You can look at the goal\nof a GC as fixing that “glitch” while sacrificing as little throughput as\npossible.

    \n
    • \n
\n\n
    \n
  • \n

    Latency is the longest continuous chunk of time where the user’s\nprogram is completely paused while garbage collection happens. It’s a\nmeasure of how “chunky” the collector is. Latency is an entirely different\nmetric than throughput.

    \n

    Consider two runs of a clox program that both take ten seconds. In the first\nrun, the GC kicks in once and spends a solid second in collectGarbage() in\none massive collection. In the second run, the GC gets invoked five times,\neach for a fifth of a second. The total amount of time spent collecting is\nstill a second, so the throughput is 90% in both cases. But in the second\nrun, the latency is only 1/5th of a second, five times less than in the\nfirst.

    \n
    • \n
\n

\"A\n\n

If you like analogies, imagine your program is a bakery selling fresh-baked\nbread to customers. Throughput is the total number of warm, crusty baguettes you\ncan serve to customers in a single day. Latency is how long the unluckiest\ncustomer has to wait in line before they get served.

\n

Running the garbage collector is like shutting\ndown the bakery temporarily to go through all of the dishes, sort out the dirty\nfrom the clean, and then wash the used ones. In our analogy, we don’t have\ndedicated dishwashers, so while this is going on, no baking is happening. The\nbaker is washing up.

\n\n

Selling fewer loaves of bread a day is bad, and making any particular customer\nsit and wait while you clean all the dishes is too. The goal is to maximize\nthroughput and minimize latency, but there is no free lunch, even inside a\nbakery. Garbage collectors make different trade-offs between how much throughput\nthey sacrifice and latency they tolerate.

\n

Being able to make these trade-offs is useful because different user programs\nhave different needs. An overnight batch job that is generating a report from a\nterabyte of data just needs to get as much work done as fast as possible.\nThroughput is queen. Meanwhile, an app running on a user’s smartphone needs to\nalways respond immediately to user input so that dragging on the screen feels\nbuttery smooth. The app can’t freeze for a few\nseconds while the GC mucks around in the heap.

\n\n

As a garbage collector author, you control some of the trade-off between\nthroughput and latency by your choice of collection algorithm. But even within a\nsingle algorithm, we have a lot of control over how frequently the collector\nruns.

\n

Our collector is a stop-the-world GC which\nmeans the user’s program is paused until the entire garbage collection process\nhas completed. If we wait a long time before we run the collector, then a large\nnumber of dead objects will accumulate. That leads to a very long pause while\nthe collector runs, and thus high latency. So, clearly, we want to run the\ncollector really frequently.

\n\n

But every time the collector runs, it spends some time visiting live objects.\nThat doesn’t really do anything useful (aside from ensuring that they don’t\nincorrectly get deleted). Time visiting live objects is time not freeing memory\nand also time not running user code. If you run the GC really frequently, then\nthe user’s program doesn’t have enough time to even generate new garbage for the\nVM to collect. The VM will spend all of its time obsessively revisiting the same\nset of live objects over and over, and throughput will suffer. So, clearly, we\nwant to run the collector really infrequently.

\n

In fact, we want something in the middle, and the frequency of when the\ncollector runs is one of our main knobs for tuning the trade-off between latency\nand throughput.

\n

26 . 6 . 2Self-adjusting heap

\n

We want our GC to run frequently enough to minimize latency but infrequently\nenough to maintain decent throughput. But how do we find the balance between\nthese when we have no idea how much memory the user’s program needs and how\noften it allocates? We could pawn the problem onto the user and force them to\npick by exposing GC tuning parameters. Many VMs do this. But if we, the GC\nauthors, don’t know how to tune it well, odds are good most users won’t either.\nThey deserve a reasonable default behavior.

\n

I’ll be honest with you, this is not my area of expertise. I’ve talked to a\nnumber of professional GC hackers—this is something you can build an entire\ncareer on—and read a lot of the literature, and all of the answers I got\nwere . . . vague. The strategy I ended up picking is common, pretty simple, and (I\nhope!) good enough for most uses.

\n

The idea is that the collector frequency automatically adjusts based on the live\nsize of the heap. We track the total number of bytes of managed memory that the\nVM has allocated. When it goes above some threshold, we trigger a GC. After\nthat, we note how many bytes of memory remain—how many were not freed. Then\nwe adjust the threshold to some value larger than that.

\n

The result is that as the amount of live memory increases, we collect less\nfrequently in order to avoid sacrificing throughput by re-traversing the growing\npile of live objects. As the amount of live memory goes down, we collect more\nfrequently so that we don’t lose too much latency by waiting too long.

\n

The implementation requires two new bookkeeping fields in the VM.

\n
  ObjUpvalue* openUpvalues;\n
vm.h
\nin struct VM
\n
\n\n  size_t bytesAllocated;\n  size_t nextGC;\n
  Obj* objects;\n
\n
vm.h, in struct VM
\n\n

The first is a running total of the number of bytes of managed memory the VM has\nallocated. The second is the threshold that triggers the next collection. We\ninitialize them when the VM starts up.

\n
  vm.objects = NULL;\n
vm.c
\nin initVM()
\n
  vm.bytesAllocated = 0;\n  vm.nextGC = 1024 * 1024;\n
\n\n  vm.grayCount = 0;\n
\n
vm.c, in initVM()
\n\n

The starting threshold here is arbitrary. It’s similar\nto the initial capacity we picked for our various dynamic arrays. The goal is to\nnot trigger the first few GCs too quickly but also to not wait too long. If we\nhad some real-world Lox programs, we could profile those to tune this. But since\nall we have are toy programs, I just picked a number.

\n\n

Every time we allocate or free some memory, we adjust the counter by that delta.

\n
void* reallocate(void* pointer, size_t oldSize, size_t newSize) {\n
memory.c
\nin reallocate()
\n
  vm.bytesAllocated += newSize - oldSize;\n
  if (newSize > oldSize) {\n
\n
memory.c, in reallocate()
\n\n

When the total crosses the limit, we run the collector.

\n
    collectGarbage();\n#endif\n
memory.c
\nin reallocate()
\n
\n\n    if (vm.bytesAllocated > vm.nextGC) {\n      collectGarbage();\n    }\n
  }\n
\n
memory.c, in reallocate()
\n\n

Now, finally, our garbage collector actually does something when the user runs a\nprogram without our hidden diagnostic flag enabled. The sweep phase frees\nobjects by calling reallocate(), which lowers the value of bytesAllocated,\nso after the collection completes, we know how many live bytes remain. We adjust\nthe threshold of the next GC based on that.

\n
  sweep();\n
memory.c
\nin collectGarbage()
\n
\n\n  vm.nextGC = vm.bytesAllocated * GC_HEAP_GROW_FACTOR;\n
\n\n#ifdef DEBUG_LOG_GC\n
\n
memory.c, in collectGarbage()
\n\n

The threshold is a multiple of the heap size. This way, as the amount of memory\nthe program uses grows, the threshold moves farther out to limit the total time\nspent re-traversing the larger live set. Like other numbers in this chapter, the\nscaling factor is basically arbitrary.

\n
#endif\n
memory.c
\n
\n\n#define GC_HEAP_GROW_FACTOR 2\n
\n\nvoid* reallocate(void* pointer, size_t oldSize, size_t newSize) {\n
\n
memory.c
\n\n

You’d want to tune this in your implementation once you had some real programs\nto benchmark it on. Right now, we can at least log some of the statistics that\nwe have. We capture the heap size before the collection.

\n
  printf("-- gc begin\\n");\n
memory.c
\nin collectGarbage()
\n
  size_t before = vm.bytesAllocated;\n
#endif\n
\n
memory.c, in collectGarbage()
\n\n

And then print the results at the end.

\n
  printf("-- gc end\\n");\n
memory.c
\nin collectGarbage()
\n
  printf("   collected %zu bytes (from %zu to %zu) next at %zu\\n",\n         before - vm.bytesAllocated, before, vm.bytesAllocated,\n         vm.nextGC);\n
#endif\n
\n
memory.c, in collectGarbage()
\n\n

This way we can see how much the garbage collector accomplished while it ran.

\n

26 . 7Garbage Collection Bugs

\n

In theory, we are all done now. We have a GC. It kicks in periodically, collects\nwhat it can, and leaves the rest. If this were a typical textbook, we would wipe\nthe dust from our hands and bask in the soft glow of the flawless marble edifice\nwe have created.

\n

But I aim to teach you not just the theory of programming languages but the\nsometimes painful reality. I am going to roll over a rotten log and show you the\nnasty bugs that live under it, and garbage collector bugs really are some of the\ngrossest invertebrates out there.

\n

The collector’s job is to free dead objects and preserve live ones. Mistakes are\neasy to make in both directions. If the VM fails to free objects that aren’t\nneeded, it slowly leaks memory. If it frees an object that is in use, the user’s\nprogram can access invalid memory. These failures often don’t immediately cause\na crash, which makes it hard for us to trace backward in time to find the bug.

\n

This is made harder by the fact that we don’t know when the collector will run.\nAny call that eventually allocates some memory is a place in the VM where a\ncollection could happen. It’s like musical chairs. At any point, the GC might\nstop the music. Every single heap-allocated object that we want to keep needs to\nfind a chair quickly—get marked as a root or stored as a reference in some\nother object—before the sweep phase comes to kick it out of the game.

\n

How is it possible for the VM to use an object later—one that the GC itself\ndoesn’t see? How can the VM find it? The most common answer is through a pointer\nstored in some local variable on the C stack. The GC walks the VM’s value and\nCallFrame stacks, but the C stack is hidden to it.

\n\n

In previous chapters, we wrote seemingly pointless code that pushed an object\nonto the VM’s value stack, did a little work, and then popped it right back off.\nMost times, I said this was for the GC’s benefit. Now you see why. The code\nbetween pushing and popping potentially allocates memory and thus can trigger a\nGC. We had to make sure the object was on the value stack so that the\ncollector’s mark phase would find it and keep it alive.

\n

I wrote the entire clox implementation before splitting it into chapters and\nwriting the prose, so I had plenty of time to find all of these corners and\nflush out most of these bugs. The stress testing code we put in at the beginning\nof this chapter and a pretty good test suite were very helpful.

\n

But I fixed only most of them. I left a couple in because I want to give you a\nhint of what it’s like to encounter these bugs in the wild. If you enable the\nstress test flag and run some toy Lox programs, you can probably stumble onto a\nfew. Give it a try and see if you can fix any yourself.

\n

26 . 7 . 1Adding to the constant table

\n

You are very likely to hit the first bug. The constant table each chunk owns is\na dynamic array. When the compiler adds a new constant to the current function’s\ntable, that array may need to grow. The constant itself may also be some\nheap-allocated object like a string or a nested function.

\n

The new object being added to the constant table is passed to addConstant().\nAt that moment, the object can be found only in the parameter to that function\non the C stack. That function appends the object to the constant table. If the\ntable doesn’t have enough capacity and needs to grow, it calls reallocate().\nThat in turn triggers a GC, which fails to mark the new constant object and\nthus sweeps it right before we have a chance to add it to the table. Crash.

\n

The fix, as you’ve seen in other places, is to push the constant onto the stack\ntemporarily.

\n
int addConstant(Chunk* chunk, Value value) {\n
chunk.c
\nin addConstant()
\n
  push(value);\n
  writeValueArray(&chunk->constants, value);\n
\n
chunk.c, in addConstant()
\n\n

Once the constant table contains the object, we pop it off the stack.

\n
  writeValueArray(&chunk->constants, value);\n
chunk.c
\nin addConstant()
\n
  pop();\n
  return chunk->constants.count - 1;\n
\n
chunk.c, in addConstant()
\n\n

When the GC is marking roots, it walks the chain of compilers and marks each of\ntheir functions, so the new constant is reachable now. We do need an include\nto call into the VM from the “chunk” module.

\n
#include "memory.h"\n
chunk.c
\n
#include "vm.h"\n
\n\nvoid initChunk(Chunk* chunk) {\n
\n
chunk.c
\n\n

26 . 7 . 2Interning strings

\n

Here’s another similar one. All strings are interned in clox, so whenever we\ncreate a new string, we also add it to the intern table. You can see where this\nis going. Since the string is brand new, it isn’t reachable anywhere. And\nresizing the string pool can trigger a collection. Again, we go ahead and stash\nthe string on the stack first.

\n
  string->chars = chars;\n  string->hash = hash;\n
object.c
\nin allocateString()
\n
\n\n  push(OBJ_VAL(string));\n
  tableSet(&vm.strings, string, NIL_VAL);\n
\n
object.c, in allocateString()
\n\n

And then pop it back off once it’s safely nestled in the table.

\n
  tableSet(&vm.strings, string, NIL_VAL);\n
object.c
\nin allocateString()
\n
  pop();\n\n
  return string;\n}\n
\n
object.c, in allocateString()
\n\n

This ensures the string is safe while the table is being resized. Once it\nsurvives that, allocateString() will return it to some caller which can then\ntake responsibility for ensuring the string is still reachable before the next\nheap allocation occurs.

\n

26 . 7 . 3Concatenating strings

\n

One last example: Over in the interpreter, the OP_ADD instruction can be used\nto concatenate two strings. As it does with numbers, it pops the two operands\nfrom the stack, computes the result, and pushes that new value back onto the\nstack. For numbers that’s perfectly safe.

\n

But concatenating two strings requires allocating a new character array on the\nheap, which can in turn trigger a GC. Since we’ve already popped the operand\nstrings by that point, they can potentially be missed by the mark phase and get\nswept away. Instead of popping them off the stack eagerly, we peek them.

\n
static void concatenate() {\n
vm.c
\nin concatenate()
\nreplace 2 lines
\n
  ObjString* b = AS_STRING(peek(0));\n  ObjString* a = AS_STRING(peek(1));\n
\n\n  int length = a->length + b->length;\n
\n
vm.c, in concatenate(), replace 2 lines
\n\n

That way, they are still hanging out on the stack when we create the result\nstring. Once that’s done, we can safely pop them off and replace them with the\nresult.

\n
  ObjString* result = takeString(chars, length);\n
vm.c
\nin concatenate()
\n
  pop();\n  pop();\n
  push(OBJ_VAL(result));\n
\n
vm.c, in concatenate()
\n\n

Those were all pretty easy, especially because I showed you where the fix was.\nIn practice, finding them is the hard part. All you see is an object that\nshould be there but isn’t. It’s not like other bugs where you’re looking for\nthe code that causes some problem. You’re looking for the absence of code\nwhich fails to prevent a problem, and that’s a much harder search.

\n

But, for now at least, you can rest easy. As far as I know, we’ve found all of\nthe collection bugs in clox, and now we have a working, robust, self-tuning,\nmark-sweep garbage collector.

\n
\n

Challenges

\n
    \n
  1. \n

    The Obj header struct at the top of each object now has three fields:\ntype, isMarked, and next. How much memory do those take up (on your\nmachine)? Can you come up with something more compact? Is there a runtime\ncost to doing so?

    \n
    2. \n
  2. \n

    When the sweep phase traverses a live object, it clears the isMarked\nfield to prepare it for the next collection cycle. Can you come up with a\nmore efficient approach?

    \n
    3. \n
  3. \n

    Mark-sweep is only one of a variety of garbage collection algorithms out\nthere. Explore those by replacing or augmenting the current collector with\nanother one. Good candidates to consider are reference counting, Cheney’s\nalgorithm, or the Lisp 2 mark-compact algorithm.

    \n
    4. \n
\n
\n
\n

Design Note: Generational Collectors

\n

A collector loses throughput if it spends a long time re-visiting objects that\nare still alive. But it can increase latency if it avoids collecting and\naccumulates a large pile of garbage to wade through. If only there were some way\nto tell which objects were likely to be long-lived and which weren’t. Then the\nGC could avoid revisiting the long-lived ones as often and clean up the\nephemeral ones more frequently.

\n

It turns out there kind of is. Many years ago, GC researchers gathered metrics\non the lifetime of objects in real-world running programs. They tracked every\nobject when it was allocated, and eventually when it was no longer needed, and\nthen graphed out how long objects tended to live.

\n

They discovered something they called the generational hypothesis, or the\nmuch less tactful term infant mortality. Their observation was that most\nobjects are very short-lived but once they survive beyond a certain age, they\ntend to stick around quite a long time. The longer an object has lived, the\nlonger it likely will continue to live. This observation is powerful because\nit gave them a handle on how to partition objects into groups that benefit from\nfrequent collections and those that don’t.

\n

They designed a technique called generational garbage collection. It works\nlike this: Every time a new object is allocated, it goes into a special,\nrelatively small region of the heap called the “nursery”. Since objects tend to\ndie young, the garbage collector is invoked frequently over the objects just in this region.

\n\n

Each time the GC runs over the nursery is called a “generation”. Any objects\nthat are no longer needed get freed. Those that survive are now considered one\ngeneration older, and the GC tracks this for each object. If an object survives\na certain number of generations—often just a single collection—it gets\ntenured. At this point, it is copied out of the nursery into a much larger\nheap region for long-lived objects. The garbage collector runs over that region\ntoo, but much less frequently since odds are good that most of those objects\nwill still be alive.

\n

Generational collectors are a beautiful marriage of empirical data—the\nobservation that object lifetimes are not evenly distributed—and clever\nalgorithm design that takes advantage of that fact. They’re also conceptually\nquite simple. You can think of one as just two separately tuned GCs and a pretty\nsimple policy for moving objects from one to the other.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/global-variables.html", "content": "\n\n\n\nGlobal Variables · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
21
\n

Global Variables

\n\n
\n

If only there could be an invention that bottled up a memory, like scent. And\nit never faded, and it never got stale. And then, when one wanted it, the\nbottle could be uncorked, and it would be like living the moment all over\nagain.

\n

Daphne du Maurier, Rebecca

\n
\n

The previous chapter was a long exploration of one big, deep,\nfundamental computer science data structure. Heavy on theory and concept. There\nmay have been some discussion of big-O notation and algorithms. This chapter has\nfewer intellectual pretensions. There are no large ideas to learn. Instead, it’s\na handful of straightforward engineering tasks. Once we’ve completed them, our\nvirtual machine will support variables.

\n

Actually, it will support only global variables. Locals are coming in the\nnext chapter. In jlox, we managed to cram them both into a single chapter\nbecause we used the same implementation technique for all variables. We built a\nchain of environments, one for each scope, all the way up to the top. That was a\nsimple, clean way to learn how to manage state.

\n

But it’s also slow. Allocating a new hash table each time you enter a block or\ncall a function is not the road to a fast VM. Given how much code is concerned\nwith using variables, if variables go slow, everything goes slow. For clox,\nwe’ll improve that by using a much more efficient strategy for local variables, but globals aren’t as easily optimized.

\n\n

A quick refresher on Lox semantics: Global variables in Lox are “late bound”, or\nresolved dynamically. This means you can compile a chunk of code that refers to\na global variable before it’s defined. As long as the code doesn’t execute\nbefore the definition happens, everything is fine. In practice, that means you\ncan refer to later variables inside the body of functions.

\n
fun showVariable() {\n  print global;\n}\n\nvar global = "after";\nshowVariable();\n
\n

Code like this might seem odd, but it’s handy for defining mutually recursive\nfunctions. It also plays nicer with the REPL. You can write a little function in\none line, then define the variable it uses in the next.

\n

Local variables work differently. Since a local variable’s declaration always\noccurs before it is used, the VM can resolve them at compile time, even in a\nsimple single-pass compiler. That will let us use a smarter representation for\nlocals. But that’s for the next chapter. Right now, let’s just worry about\nglobals.

\n

21 . 1Statements

\n

Variables come into being using variable declarations, which means now is also\nthe time to add support for statements to our compiler. If you recall, Lox\nsplits statements into two categories. “Declarations” are those statements that\nbind a new name to a value. The other kinds of statements—control flow,\nprint, etc.—are just called “statements”. We disallow declarations directly\ninside control flow statements, like this:

\n
if (monday) var croissant = "yes"; // Error.\n
\n

Allowing it would raise confusing questions around the scope of the variable.\nSo, like other languages, we prohibit it syntactically by having a separate\ngrammar rule for the subset of statements that are allowed inside a control\nflow body.

\n
statement      → exprStmt\n               | forStmt\n               | ifStmt\n               | printStmt\n               | returnStmt\n               | whileStmt\n               | block ;\n
\n

Then we use a separate rule for the top level of a script and inside a block.

\n
declaration    → classDecl\n               | funDecl\n               | varDecl\n               | statement ;\n
\n

The declaration rule contains the statements that declare names, and also\nincludes statement so that all statement types are allowed. Since block\nitself is in statement, you can put declarations inside a control flow construct by nesting them inside a\nblock.

\n\n

In this chapter, we’ll cover only a couple of statements and one\ndeclaration.

\n
statement      → exprStmt\n               | printStmt ;\n\ndeclaration    → varDecl\n               | statement ;\n
\n

Up to now, our VM considered a “program” to be a single expression since that’s\nall we could parse and compile. In a full Lox implementation, a program is a\nsequence of declarations. We’re ready to support that now.

\n
  advance();\n
compiler.c
\nin compile()
\nreplace 2 lines
\n
\n\n  while (!match(TOKEN_EOF)) {\n    declaration();\n  }\n\n
  endCompiler();\n
\n
compiler.c, in compile(), replace 2 lines
\n\n

We keep compiling declarations until we hit the end of the source file. We\ncompile a single declaration using this:

\n
compiler.c
\nadd after expression()
\n
static void declaration() {\n  statement();\n}\n
\n
compiler.c, add after expression()
\n\n

We’ll get to variable declarations later in the chapter, so for now, we simply\nforward to statement().

\n
compiler.c
\nadd after declaration()
\n
static void statement() {\n  if (match(TOKEN_PRINT)) {\n    printStatement();\n  }\n}\n
\n
compiler.c, add after declaration()
\n\n

Blocks can contain declarations, and control flow statements can contain other\nstatements. That means these two functions will eventually be recursive. We may\nas well write out the forward declarations now.

\n
static void expression();\n
compiler.c
\nadd after expression()
\n
static void statement();\nstatic void declaration();\n
static ParseRule* getRule(TokenType type);\n
\n
compiler.c, add after expression()
\n\n

21 . 1 . 1Print statements

\n

We have two statement types to support in this chapter. Let’s start with print\nstatements, which begin, naturally enough, with a print token. We detect that\nusing this helper function:

\n
compiler.c
\nadd after consume()
\n
static bool match(TokenType type) {\n  if (!check(type)) return false;\n  advance();\n  return true;\n}\n
\n
compiler.c, add after consume()
\n\n

You may recognize it from jlox. If the current token has the given type, we\nconsume the token and return true. Otherwise we leave the token alone and\nreturn false. This helper function is implemented\nin terms of this other helper:

\n\n
compiler.c
\nadd after consume()
\n
static bool check(TokenType type) {\n  return parser.current.type == type;\n}\n
\n
compiler.c, add after consume()
\n\n

The check() function returns true if the current token has the given type.\nIt seems a little silly to wrap this in a function, but\nwe’ll use it more later, and I think short verb-named functions like this make\nthe parser easier to read.

\n\n

If we did match the print token, then we compile the rest of the statement\nhere:

\n
compiler.c
\nadd after expression()
\n
static void printStatement() {\n  expression();\n  consume(TOKEN_SEMICOLON, "Expect ';' after value.");\n  emitByte(OP_PRINT);\n}\n
\n
compiler.c, add after expression()
\n\n

A print statement evaluates an expression and prints the result, so we first\nparse and compile that expression. The grammar expects a semicolon after that,\nso we consume it. Finally, we emit a new instruction to print the result.

\n
  OP_NEGATE,\n
chunk.h
\nin enum OpCode
\n
  OP_PRINT,\n
  OP_RETURN,\n
\n
chunk.h, in enum OpCode
\n\n

At runtime, we execute this instruction like so:

\n
        break;\n
vm.c
\nin run()
\n
      case OP_PRINT: {\n        printValue(pop());\n        printf("\\n");\n        break;\n      }\n
      case OP_RETURN: {\n
\n
vm.c, in run()
\n\n

When the interpreter reaches this instruction, it has already executed the code\nfor the expression, leaving the result value on top of the stack. Now we simply\npop and print it.

\n

Note that we don’t push anything else after that. This is a key difference\nbetween expressions and statements in the VM. Every bytecode instruction has a\nstack effect that describes how the instruction\nmodifies the stack. For example, OP_ADD pops two values and pushes one,\nleaving the stack one element smaller than before.

\n\n

You can sum the stack effects of a series of instructions to get their total\neffect. When you add the stack effects of the series of instructions compiled\nfrom any complete expression, it will total one. Each expression leaves one\nresult value on the stack.

\n

The bytecode for an entire statement has a total stack effect of zero. Since a\nstatement produces no values, it ultimately leaves the stack unchanged, though\nit of course uses the stack while it’s doing its thing. This is important\nbecause when we get to control flow and looping, a program might execute a long\nseries of statements. If each statement grew or shrank the stack, it might\neventually overflow or underflow.

\n

While we’re in the interpreter loop, we should delete a bit of code.

\n
      case OP_RETURN: {\n
vm.c
\nin run()
\nreplace 2 lines
\n
        // Exit interpreter.\n
        return INTERPRET_OK;\n
\n
vm.c, in run(), replace 2 lines
\n\n

When the VM only compiled and evaluated a single expression, we had some\ntemporary code in OP_RETURN to output the value. Now that we have statements\nand print, we don’t need that anymore. We’re one step closer to the complete implementation of clox.

\n\n

As usual, a new instruction needs support in the disassembler.

\n
      return simpleInstruction("OP_NEGATE", offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_PRINT:\n      return simpleInstruction("OP_PRINT", offset);\n
    case OP_RETURN:\n
\n
debug.c, in disassembleInstruction()
\n\n

That’s our print statement. If you want, give it a whirl:

\n
print 1 + 2;\nprint 3 * 4;\n
\n

Exciting! OK, maybe not thrilling, but we can build scripts that contain as many\nstatements as we want now, which feels like progress.

\n

21 . 1 . 2Expression statements

\n

Wait until you see the next statement. If we don’t see a print keyword, then\nwe must be looking at an expression statement.

\n
    printStatement();\n
compiler.c
\nin statement()
\n
  } else {\n    expressionStatement();\n
  }\n
\n
compiler.c, in statement()
\n\n

It’s parsed like so:

\n
compiler.c
\nadd after expression()
\n
static void expressionStatement() {\n  expression();\n  consume(TOKEN_SEMICOLON, "Expect ';' after expression.");\n  emitByte(OP_POP);\n}\n
\n
compiler.c, add after expression()
\n\n

An “expression statement” is simply an expression followed by a semicolon.\nThey’re how you write an expression in a context where a statement is expected.\nUsually, it’s so that you can call a function or evaluate an assignment for its\nside effect, like this:

\n
brunch = "quiche";\neat(brunch);\n
\n

Semantically, an expression statement evaluates the expression and discards the\nresult. The compiler directly encodes that behavior. It compiles the expression,\nand then emits an OP_POP instruction.

\n
  OP_FALSE,\n
chunk.h
\nin enum OpCode
\n
  OP_POP,\n
  OP_EQUAL,\n
\n
chunk.h, in enum OpCode
\n\n

As the name implies, that instruction pops the top value off the stack and\nforgets it.

\n
      case OP_FALSE: push(BOOL_VAL(false)); break;\n
vm.c
\nin run()
\n
      case OP_POP: pop(); break;\n
      case OP_EQUAL: {\n
\n
vm.c, in run()
\n\n

We can disassemble it too.

\n
      return simpleInstruction("OP_FALSE", offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_POP:\n      return simpleInstruction("OP_POP", offset);\n
    case OP_EQUAL:\n
\n
debug.c, in disassembleInstruction()
\n\n

Expression statements aren’t very useful yet since we can’t create any\nexpressions that have side effects, but they’ll be essential when we\nadd functions later. The majority of\nstatements in real-world code in languages like C are expression statements.

\n\n

21 . 1 . 3Error synchronization

\n

While we’re getting this initial work done in the compiler, we can tie off a\nloose end we left several chapters back. Like jlox, clox uses panic\nmode error recovery to minimize the number of cascaded compile errors that it\nreports. The compiler exits panic mode when it reaches a synchronization point.\nFor Lox, we chose statement boundaries as that point. Now that we have\nstatements, we can implement synchronization.

\n
  statement();\n
compiler.c
\nin declaration()
\n
\n\n  if (parser.panicMode) synchronize();\n
}\n
\n
compiler.c, in declaration()
\n\n

If we hit a compile error while parsing the previous statement, we enter panic\nmode. When that happens, after the statement we start synchronizing.

\n
compiler.c
\nadd after printStatement()
\n
static void synchronize() {\n  parser.panicMode = false;\n\n  while (parser.current.type != TOKEN_EOF) {\n    if (parser.previous.type == TOKEN_SEMICOLON) return;\n    switch (parser.current.type) {\n      case TOKEN_CLASS:\n      case TOKEN_FUN:\n      case TOKEN_VAR:\n      case TOKEN_FOR:\n      case TOKEN_IF:\n      case TOKEN_WHILE:\n      case TOKEN_PRINT:\n      case TOKEN_RETURN:\n        return;\n\n      default:\n        ; // Do nothing.\n    }\n\n    advance();\n  }\n}\n
\n
compiler.c, add after printStatement()
\n\n

We skip tokens indiscriminately until we reach something that looks like a\nstatement boundary. We recognize the boundary by looking for a preceding token\nthat can end a statement, like a semicolon. Or we’ll look for a subsequent token\nthat begins a statement, usually one of the control flow or declaration\nkeywords.

\n

21 . 2Variable Declarations

\n

Merely being able to print doesn’t win your language any prizes at the\nprogramming language fair, so let’s move on to\nsomething a little more ambitious and get variables going. There are three\noperations we need to support:

\n\n
    \n
  • Declaring a new variable using a var statement.
    • \n
  • Accessing the value of a variable using an identifier expression.
    • \n
  • Storing a new value in an existing variable using an assignment expression.
    • \n
\n

We can’t do either of the last two until we have some variables, so we start\nwith declarations.

\n
static void declaration() {\n
compiler.c
\nin declaration()
\nreplace 1 line
\n
  if (match(TOKEN_VAR)) {\n    varDeclaration();\n  } else {\n    statement();\n  }\n
\n\n  if (parser.panicMode) synchronize();\n
\n
compiler.c, in declaration(), replace 1 line
\n\n

The placeholder parsing function we sketched out for the declaration grammar\nrule has an actual production now. If we match a var token, we jump here:

\n
compiler.c
\nadd after expression()
\n
static void varDeclaration() {\n  uint8_t global = parseVariable("Expect variable name.");\n\n  if (match(TOKEN_EQUAL)) {\n    expression();\n  } else {\n    emitByte(OP_NIL);\n  }\n  consume(TOKEN_SEMICOLON,\n          "Expect ';' after variable declaration.");\n\n  defineVariable(global);\n}\n
\n
compiler.c, add after expression()
\n\n

The keyword is followed by the variable name. That’s compiled by\nparseVariable(), which we’ll get to in a second. Then we look for an =\nfollowed by an initializer expression. If the user doesn’t initialize the\nvariable, the compiler implicitly initializes it to nil by emitting an OP_NIL instruction. Either way, we\nexpect the statement to be terminated with a semicolon.

\n\n

There are two new functions here for working with variables and identifiers.\nHere is the first:

\n
static void parsePrecedence(Precedence precedence);\n\n
compiler.c
\nadd after parsePrecedence()
\n
static uint8_t parseVariable(const char* errorMessage) {\n  consume(TOKEN_IDENTIFIER, errorMessage);\n  return identifierConstant(&parser.previous);\n}\n
\n
compiler.c, add after parsePrecedence()
\n\n

It requires the next token to be an identifier, which it consumes and sends\nhere:

\n
static void parsePrecedence(Precedence precedence);\n\n
compiler.c
\nadd after parsePrecedence()
\n
static uint8_t identifierConstant(Token* name) {\n  return makeConstant(OBJ_VAL(copyString(name->start,\n                                         name->length)));\n}\n
\n
compiler.c, add after parsePrecedence()
\n\n

This function takes the given token and adds its lexeme to the chunk’s constant\ntable as a string. It then returns the index of that constant in the constant\ntable.

\n

Global variables are looked up by name at runtime. That means the VM—the\nbytecode interpreter loop—needs access to the name. A whole string is too big\nto stuff into the bytecode stream as an operand. Instead, we store the string in\nthe constant table and the instruction then refers to the name by its index in\nthe table.

\n

This function returns that index all the way to varDeclaration() which later\nhands it over to here:

\n
compiler.c
\nadd after parseVariable()
\n
static void defineVariable(uint8_t global) {\n  emitBytes(OP_DEFINE_GLOBAL, global);\n}\n
\n
compiler.c, add after parseVariable()
\n\n

This outputs the bytecode instruction that defines\nthe new variable and stores its initial value. The index of the variable’s name\nin the constant table is the instruction’s operand. As usual in a stack-based\nVM, we emit this instruction last. At runtime, we execute the code for the\nvariable’s initializer first. That leaves the value on the stack. Then this\ninstruction takes that value and stores it away for later.

\n\n

Over in the runtime, we begin with this new instruction:

\n
  OP_POP,\n
chunk.h
\nin enum OpCode
\n
  OP_DEFINE_GLOBAL,\n
  OP_EQUAL,\n
\n
chunk.h, in enum OpCode
\n\n

Thanks to our handy-dandy hash table, the implementation isn’t too hard.

\n
      case OP_POP: pop(); break;\n
vm.c
\nin run()
\n
      case OP_DEFINE_GLOBAL: {\n        ObjString* name = READ_STRING();\n        tableSet(&vm.globals, name, peek(0));\n        pop();\n        break;\n      }\n
      case OP_EQUAL: {\n
\n
vm.c, in run()
\n\n

We get the name of the variable from the constant table. Then we take the value from the top of the stack and store it in a\nhash table with that name as the key.

\n\n

This code doesn’t check to see if the key is already in the table. Lox is pretty\nlax with global variables and lets you redefine them without error. That’s\nuseful in a REPL session, so the VM supports that by simply overwriting the\nvalue if the key happens to already be in the hash table.

\n

There’s another little helper macro:

\n
#define READ_CONSTANT() (vm.chunk->constants.values[READ_BYTE()])\n
vm.c
\nin run()
\n
#define READ_STRING() AS_STRING(READ_CONSTANT())\n
#define BINARY_OP(valueType, op) \\\n
\n
vm.c, in run()
\n\n

It reads a one-byte operand from the bytecode chunk. It treats that as an index\ninto the chunk’s constant table and returns the string at that index. It doesn’t\ncheck that the value is a string—it just indiscriminately casts it. That’s\nsafe because the compiler never emits an instruction that refers to a non-string\nconstant.

\n

Because we care about lexical hygiene, we also undefine this macro at the end of\nthe interpret function.

\n
#undef READ_CONSTANT\n
vm.c
\nin run()
\n
#undef READ_STRING\n
#undef BINARY_OP\n
\n
vm.c, in run()
\n\n

I keep saying “the hash table”, but we don’t actually have one yet. We need a\nplace to store these globals. Since we want them to persist as long as clox is\nrunning, we store them right in the VM.

\n
  Value* stackTop;\n
vm.h
\nin struct VM
\n
  Table globals;\n
  Table strings;\n
\n
vm.h, in struct VM
\n\n

As we did with the string table, we need to initialize the hash table to a valid\nstate when the VM boots up.

\n
  vm.objects = NULL;\n
vm.c
\nin initVM()
\n
\n\n  initTable(&vm.globals);\n
  initTable(&vm.strings);\n
\n
vm.c, in initVM()
\n\n

And we tear it down when we exit.

\n\n
void freeVM() {\n
vm.c
\nin freeVM()
\n
  freeTable(&vm.globals);\n
  freeTable(&vm.strings);\n
\n
vm.c, in freeVM()
\n\n

As usual, we want to be able to disassemble the new instruction too.

\n
      return simpleInstruction("OP_POP", offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_DEFINE_GLOBAL:\n      return constantInstruction("OP_DEFINE_GLOBAL", chunk,\n                                 offset);\n
    case OP_EQUAL:\n
\n
debug.c, in disassembleInstruction()
\n\n

And with that, we can define global variables. Not that users can tell that\nthey’ve done so, because they can’t actually use them. So let’s fix that next.

\n

21 . 3Reading Variables

\n

As in every programming language ever, we access a variable’s value using its\nname. We hook up identifier tokens to the expression parser here:

\n
  [TOKEN_LESS_EQUAL]    = {NULL,     binary, PREC_COMPARISON},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_IDENTIFIER]    = {variable, NULL,   PREC_NONE},\n
  [TOKEN_STRING]        = {string,   NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n

That calls this new parser function:

\n
compiler.c
\nadd after string()
\n
static void variable() {\n  namedVariable(parser.previous);\n}\n
\n
compiler.c, add after string()
\n\n

Like with declarations, there are a couple of tiny helper functions that seem\npointless now but will become more useful in later chapters. I promise.

\n
compiler.c
\nadd after string()
\n
static void namedVariable(Token name) {\n  uint8_t arg = identifierConstant(&name);\n  emitBytes(OP_GET_GLOBAL, arg);\n}\n
\n
compiler.c, add after string()
\n\n

This calls the same identifierConstant() function from before to take the\ngiven identifier token and add its lexeme to the chunk’s constant table as a\nstring. All that remains is to emit an instruction that loads the global\nvariable with that name. Here’s the instruction:

\n
  OP_POP,\n
chunk.h
\nin enum OpCode
\n
  OP_GET_GLOBAL,\n
  OP_DEFINE_GLOBAL,\n
\n
chunk.h, in enum OpCode
\n\n

Over in the interpreter, the implementation mirrors OP_DEFINE_GLOBAL.

\n
      case OP_POP: pop(); break;\n
vm.c
\nin run()
\n
      case OP_GET_GLOBAL: {\n        ObjString* name = READ_STRING();\n        Value value;\n        if (!tableGet(&vm.globals, name, &value)) {\n          runtimeError("Undefined variable '%s'.", name->chars);\n          return INTERPRET_RUNTIME_ERROR;\n        }\n        push(value);\n        break;\n      }\n
      case OP_DEFINE_GLOBAL: {\n
\n
vm.c, in run()
\n\n

We pull the constant table index from the instruction’s operand and get the\nvariable name. Then we use that as a key to look up the variable’s value in the\nglobals hash table.

\n

If the key isn’t present in the hash table, it means that global variable has\nnever been defined. That’s a runtime error in Lox, so we report it and exit the\ninterpreter loop if that happens. Otherwise, we take the value and push it\nonto the stack.

\n
      return simpleInstruction("OP_POP", offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_GET_GLOBAL:\n      return constantInstruction("OP_GET_GLOBAL", chunk, offset);\n
    case OP_DEFINE_GLOBAL:\n
\n
debug.c, in disassembleInstruction()
\n\n

A little bit of disassembling, and we’re done. Our interpreter is now able to\nrun code like this:

\n
var beverage = "cafe au lait";\nvar breakfast = "beignets with " + beverage;\nprint breakfast;\n
\n

There’s only one operation left.

\n

21 . 4Assignment

\n

Throughout this book, I’ve tried to keep you on a fairly safe and easy path. I\ndon’t avoid hard problems, but I try to not make the solutions more complex\nthan they need to be. Alas, other design choices in our bytecode compiler make assignment annoying to implement.

\n\n

Our bytecode VM uses a single-pass compiler. It parses and generates bytecode\non the fly without any intermediate AST. As soon as it recognizes a piece of\nsyntax, it emits code for it. Assignment doesn’t naturally fit that. Consider:

\n
menu.brunch(sunday).beverage = "mimosa";\n
\n

In this code, the parser doesn’t realize menu.brunch(sunday).beverage is the\ntarget of an assignment and not a normal expression until it reaches =, many\ntokens after the first menu. By then, the compiler has already emitted\nbytecode for the whole thing.

\n

The problem is not as dire as it might seem, though. Look at how the parser sees that example:

\"The\n

Even though the .beverage part must not be compiled as a get expression,\neverything to the left of the . is an expression, with the normal expression\nsemantics. The menu.brunch(sunday) part can be compiled and executed as usual.

\n

Fortunately for us, the only semantic differences on the left side of an\nassignment appear at the very right-most end of the tokens, immediately\npreceding the =. Even though the receiver of a setter may be an arbitrarily\nlong expression, the part whose behavior differs from a get expression is only\nthe trailing identifier, which is right before the =. We don’t need much\nlookahead to realize beverage should be compiled as a set expression and not a\ngetter.

\n

Variables are even easier since they are just a single bare identifier before an\n=. The idea then is that right before compiling an expression that can also\nbe used as an assignment target, we look for a subsequent = token. If we see\none, we compile it as an assignment or setter instead of a variable access or\ngetter.

\n

We don’t have setters to worry about yet, so all we need to handle are variables.

\n
  uint8_t arg = identifierConstant(&name);\n
compiler.c
\nin namedVariable()
\nreplace 1 line
\n
\n\n  if (match(TOKEN_EQUAL)) {\n    expression();\n    emitBytes(OP_SET_GLOBAL, arg);\n  } else {\n    emitBytes(OP_GET_GLOBAL, arg);\n  }\n
}\n
\n
compiler.c, in namedVariable(), replace 1 line
\n\n

In the parse function for identifier expressions, we look for an equals sign\nafter the identifier. If we find one, instead of emitting code for a variable\naccess, we compile the assigned value and then emit an assignment instruction.

\n

That’s the last instruction we need to add in this chapter.

\n
  OP_DEFINE_GLOBAL,\n
chunk.h
\nin enum OpCode
\n
  OP_SET_GLOBAL,\n
  OP_EQUAL,\n
\n
chunk.h, in enum OpCode
\n\n

As you’d expect, its runtime behavior is similar to defining a new variable.

\n
      }\n
vm.c
\nin run()
\n
      case OP_SET_GLOBAL: {\n        ObjString* name = READ_STRING();\n        if (tableSet(&vm.globals, name, peek(0))) {\n          tableDelete(&vm.globals, name); \n          runtimeError("Undefined variable '%s'.", name->chars);\n          return INTERPRET_RUNTIME_ERROR;\n        }\n        break;\n      }\n
      case OP_EQUAL: {\n
\n
vm.c, in run()
\n\n

The main difference is what happens when the key doesn’t already exist in the\nglobals hash table. If the variable hasn’t been defined yet, it’s a runtime\nerror to try to assign to it. Lox doesn’t do implicit variable\ndeclaration.

\n\n

The other difference is that setting a variable doesn’t pop the value off the\nstack. Remember, assignment is an expression, so it needs to leave that value\nthere in case the assignment is nested inside some larger expression.

\n

Add a dash of disassembly:

\n
      return constantInstruction("OP_DEFINE_GLOBAL", chunk,\n                                 offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_SET_GLOBAL:\n      return constantInstruction("OP_SET_GLOBAL", chunk, offset);\n
    case OP_EQUAL:\n
\n
debug.c, in disassembleInstruction()
\n\n

So we’re done, right? Well . . . not quite. We’ve made a mistake! Take a gander at:

\n
a * b = c + d;\n
\n

According to Lox’s grammar, = has the lowest precedence, so this should be\nparsed roughly like:

\"The\n

Obviously, a * b isn’t a valid assignment target, so\nthis should be a syntax error. But here’s what our parser does:

\n\n
    \n
  1. First, parsePrecedence() parses a using the variable() prefix parser.
    2. \n
  2. After that, it enters the infix parsing loop.
    3. \n
  3. It reaches the * and calls binary().
    4. \n
  4. That recursively calls parsePrecedence() to parse the right-hand operand.
    5. \n
  5. That calls variable() again for parsing b.
    6. \n
  6. Inside that call to variable(), it looks for a trailing =. It sees one\nand thus parses the rest of the line as an assignment.
    7. \n
\n

In other words, the parser sees the above code like:

\"The\n

We’ve messed up the precedence handling because variable() doesn’t take into\naccount the precedence of the surrounding expression that contains the variable.\nIf the variable happens to be the right-hand side of an infix operator, or the\noperand of a unary operator, then that containing expression is too high\nprecedence to permit the =.

\n

To fix this, variable() should look for and consume the = only if it’s in\nthe context of a low-precedence expression. The code that knows the current\nprecedence is, logically enough, parsePrecedence(). The variable() function\ndoesn’t need to know the actual level. It just cares that the precedence is low\nenough to allow assignment, so we pass that fact in as a Boolean.

\n
    error("Expect expression.");\n    return;\n  }\n\n
compiler.c
\nin parsePrecedence()
\nreplace 1 line
\n
  bool canAssign = precedence <= PREC_ASSIGNMENT;\n  prefixRule(canAssign);\n
\n\n  while (precedence <= getRule(parser.current.type)->precedence) {\n
\n
compiler.c, in parsePrecedence(), replace 1 line
\n\n

Since assignment is the lowest-precedence expression, the only time we allow an\nassignment is when parsing an assignment expression or top-level expression like\nin an expression statement. That flag makes its way to the parser function here:

\n
compiler.c
\nfunction variable()
\nreplace 3 lines
\n
static void variable(bool canAssign) {\n  namedVariable(parser.previous, canAssign);\n}\n
\n
compiler.c, function variable(), replace 3 lines
\n\n

Which passes it through a new parameter:

\n
compiler.c
\nfunction namedVariable()
\nreplace 1 line
\n
static void namedVariable(Token name, bool canAssign) {\n
  uint8_t arg = identifierConstant(&name);\n
\n
compiler.c, function namedVariable(), replace 1 line
\n\n

And then finally uses it here:

\n
  uint8_t arg = identifierConstant(&name);\n\n
compiler.c
\nin namedVariable()
\nreplace 1 line
\n
  if (canAssign && match(TOKEN_EQUAL)) {\n
    expression();\n
\n
compiler.c, in namedVariable(), replace 1 line
\n\n

That’s a lot of plumbing to get literally one bit of data to the right place in\nthe compiler, but arrived it has. If the variable is nested inside some\nexpression with higher precedence, canAssign will be false and this will\nignore the = even if there is one there. Then namedVariable() returns, and\nexecution eventually makes its way back to parsePrecedence().

\n

Then what? What does the compiler do with our broken example from before? Right\nnow, variable() won’t consume the =, so that will be the current token. The\ncompiler returns back to parsePrecedence() from the variable() prefix parser\nand then tries to enter the infix parsing loop. There is no parsing function\nassociated with =, so it skips that loop.

\n

Then parsePrecedence() silently returns back to the caller. That also isn’t\nright. If the = doesn’t get consumed as part of the expression, nothing else\nis going to consume it. It’s an error and we should report it.

\n
    infixRule();\n  }\n
compiler.c
\nin parsePrecedence()
\n
\n\n  if (canAssign && match(TOKEN_EQUAL)) {\n    error("Invalid assignment target.");\n  }\n
}\n
\n
compiler.c, in parsePrecedence()
\n\n

With that, the previous bad program correctly gets an error at compile time. OK,\nnow are we done? Still not quite. See, we’re passing an argument to one of the\nparse functions. But those functions are stored in a table of function pointers,\nso all of the parse functions need to have the same type. Even though most parse\nfunctions don’t support being used as an assignment target—setters are the\nonly other one—our friendly C compiler requires\nthem all to accept the parameter.

\n\n

So we’re going to finish off this chapter with some grunt work. First, let’s go\nahead and pass the flag to the infix parse functions.

\n
    ParseFn infixRule = getRule(parser.previous.type)->infix;\n
compiler.c
\nin parsePrecedence()
\nreplace 1 line
\n
    infixRule(canAssign);\n
  }\n
\n
compiler.c, in parsePrecedence(), replace 1 line
\n\n

We’ll need that for setters eventually. Then we’ll fix the typedef for the\nfunction type.

\n
} Precedence;\n\n
compiler.c
\nadd after enum Precedence
\nreplace 1 line
\n
typedef void (*ParseFn)(bool canAssign);\n
\n\ntypedef struct {\n
\n
compiler.c, add after enum Precedence, replace 1 line
\n\n

And some completely tedious code to accept this parameter in all of our existing\nparse functions. Here:

\n
compiler.c
\nfunction binary()
\nreplace 1 line
\n
static void binary(bool canAssign) {\n
  TokenType operatorType = parser.previous.type;\n
\n
compiler.c, function binary(), replace 1 line
\n\n

And here:

\n
compiler.c
\nfunction literal()
\nreplace 1 line
\n
static void literal(bool canAssign) {\n
  switch (parser.previous.type) {\n
\n
compiler.c, function literal(), replace 1 line
\n\n

And here:

\n
compiler.c
\nfunction grouping()
\nreplace 1 line
\n
static void grouping(bool canAssign) {\n
  expression();\n
\n
compiler.c, function grouping(), replace 1 line
\n\n

And here:

\n
compiler.c
\nfunction number()
\nreplace 1 line
\n
static void number(bool canAssign) {\n
  double value = strtod(parser.previous.start, NULL);\n
\n
compiler.c, function number(), replace 1 line
\n\n

And here too:

\n
compiler.c
\nfunction string()
\nreplace 1 line
\n
static void string(bool canAssign) {\n
  emitConstant(OBJ_VAL(copyString(parser.previous.start + 1,\n
\n
compiler.c, function string(), replace 1 line
\n\n

And, finally:

\n
compiler.c
\nfunction unary()
\nreplace 1 line
\n
static void unary(bool canAssign) {\n
  TokenType operatorType = parser.previous.type;\n
\n
compiler.c, function unary(), replace 1 line
\n\n

Phew! We’re back to a C program we can compile. Fire it up and now you can run\nthis:

\n
var breakfast = "beignets";\nvar beverage = "cafe au lait";\nbreakfast = "beignets with " + beverage;\n\nprint breakfast;\n
\n

It’s starting to look like real code for an actual language!

\n
\n

Challenges

\n
    \n
  1. \n

    The compiler adds a global variable’s name to the constant table as a string\nevery time an identifier is encountered. It creates a new constant each\ntime, even if that variable name is already in a previous slot in the\nconstant table. That’s wasteful in cases where the same variable is\nreferenced multiple times by the same function. That, in turn, increases the\nodds of filling up the constant table and running out of slots since we\nallow only 256 constants in a single chunk.

    \n

    Optimize this. How does your optimization affect the performance of the\ncompiler compared to the runtime? Is this the right trade-off?

    \n
    2. \n
  2. \n

    Looking up a global variable by name in a hash table each time it is used\nis pretty slow, even with a good hash table. Can you come up with a more\nefficient way to store and access global variables without changing the\nsemantics?

    \n
    3. \n
  3. \n

    When running in the REPL, a user might write a function that references an\nunknown global variable. Then, in the next line, they declare the variable.\nLox should handle this gracefully by not reporting an “unknown variable”\ncompile error when the function is first defined.

    \n

    But when a user runs a Lox script, the compiler has access to the full\ntext of the entire program before any code is run. Consider this program:

    \n
    fun useVar() {\n  print oops;\n}\n\nvar ooops = "too many o's!";\n
    \n

    Here, we can tell statically that oops will not be defined because there\nis no declaration of that global anywhere in the program. Note that\nuseVar() is never called either, so even though the variable isn’t\ndefined, no runtime error will occur because it’s never used either.

    \n

    We could report mistakes like this as compile errors, at least when running\nfrom a script. Do you think we should? Justify your answer. What do other\nscripting languages you know do?

    \n
    4. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/hash-tables.html", "content": "\n\n\n\nHash Tables · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
20
\n

Hash Tables

\n\n
\n

Hash, x. There is no definition for this word—nobody knows what hash is.

\n

Ambrose Bierce, The Unabridged Devil’s Dictionary

\n
\n

Before we can add variables to our burgeoning virtual machine, we need some way\nto look up a value given a variable’s name. Later, when we add classes, we’ll\nalso need a way to store fields on instances. The perfect data structure for\nthese problems and others is a hash table.

\n

You probably already know what a hash table is, even if you don’t know it by\nthat name. If you’re a Java programmer, you call them “HashMaps”. C# and Python\nusers call them “dictionaries”. In C++, it’s an “unordered map”. “Objects” in\nJavaScript and “tables” in Lua are hash tables under the hood, which is what\ngives them their flexibility.

\n

A hash table, whatever your language calls it, associates a set of keys with\na set of values. Each key/value pair is an entry in the table. Given a\nkey, you can look up its corresponding value. You can add new key/value pairs\nand remove entries by key. If you add a new value for an existing key, it\nreplaces the previous entry.

\n

Hash tables appear in so many languages because they are incredibly powerful.\nMuch of this power comes from one metric: given a key, a hash table returns the\ncorresponding value in constant time, regardless\nof how many keys are in the hash table.

\n\n

That’s pretty remarkable when you think about it. Imagine you’ve got a big stack\nof business cards and I ask you to find a certain person. The bigger the pile\nis, the longer it will take. Even if the pile is nicely sorted and you’ve got\nthe manual dexterity to do a binary search by hand, you’re still talking\nO(log n). But with a hash table, it takes the\nsame time to find that business card when the stack has ten cards as when it has\na million.

\n\n

20 . 1An Array of Buckets

\n

A complete, fast hash table has a couple of moving parts. I’ll introduce them\none at a time by working through a couple of toy problems and their solutions.\nEventually, we’ll build up to a data structure that can associate any set of\nnames with their values.

\n

For now, imagine if Lox was a lot more restricted in variable names. What if a\nvariable’s name could only be a single lowercase\nletter. How could we very efficiently represent a set of variable names and\ntheir values?

\n\n

With only 26 possible variables (27 if you consider underscore a “letter”, I\nguess), the answer is easy. Declare a fixed-size array with 26 elements. We’ll\nfollow tradition and call each element a bucket. Each represents a variable\nwith a starting at index zero. If there’s a value in the array at some\nletter’s index, then that key is present with that value. Otherwise, the bucket\nis empty and that key/value pair isn’t in the data structure.

\n\n

Memory usage is great—just a single, reasonably sized array. There’s some waste from the empty buckets, but it’s\nnot huge. There’s no overhead for node pointers, padding, or other stuff you’d\nget with something like a linked list or tree.

\n

Performance is even better. Given a variable name—its character—you can\nsubtract the ASCII value of a and use the result to index directly into the\narray. Then you can either look up the existing value or store a new value\ndirectly into that slot. It doesn’t get much faster than that.

\n

This is sort of our Platonic ideal data structure. Lightning fast, dead simple,\nand compact in memory. As we add support for more complex keys, we’ll have to\nmake some concessions, but this is what we’re aiming for. Even once you add in\nhash functions, dynamic resizing, and collision resolution, this is still the\ncore of every hash table out there—a contiguous array of buckets that you\nindex directly into.

\n

20 . 1 . 1Load factor and wrapped keys

\n

Confining Lox to single-letter variables would make our job as implementers\neasier, but it’s probably no fun programming in a language that gives you only\n26 storage locations. What if we loosened it a little and allowed variables up\nto eight characters long?

\n\n

That’s small enough that we can pack all eight characters into a 64-bit integer\nand easily turn the string into a number. We can then use it as an array index.\nOr, at least, we could if we could somehow allocate a 295,148 petabyte array.\nMemory’s gotten cheaper over time, but not quite that cheap. Even if we could\nmake an array that big, it would be heinously wasteful. Almost every bucket\nwould be empty unless users started writing way bigger Lox programs than we’ve\nanticipated.

\n

Even though our variable keys cover the full 64-bit numeric range, we clearly\ndon’t need an array that large. Instead, we allocate an array with more than\nenough capacity for the entries we need, but not unreasonably large. We map the\nfull 64-bit keys down to that smaller range by taking the value modulo the size\nof the array. Doing that essentially folds the larger numeric range onto itself\nuntil it fits the smaller range of array elements.

\n

For example, say we want to store “bagel”. We allocate an array with eight\nelements, plenty enough to store it and more later. We treat the key string as a\n64-bit integer. On a little-endian machine like Intel, packing those characters\ninto a 64-bit word puts the first letter, “b” (ASCII value 98), in the\nleast-significant byte. We take that integer modulo the array size (8) to fit it in the bounds and get a bucket index, 2.\nThen we store the value there as usual.

\n\n

Using the array size as a modulus lets us map the key’s numeric range down to\nfit an array of any size. We can thus control the number of buckets\nindependently of the key range. That solves our waste problem, but introduces a\nnew one. Any two variables whose key number has the same remainder when divided\nby the array size will end up in the same bucket. Keys can collide. For\nexample, if we try to add “jam”, it also ends up in bucket 2.

\"'Bagel'\n

We have some control over this by tuning the array size. The bigger the array,\nthe fewer the indexes that get mapped to the same bucket and the fewer the\ncollisions that are likely to occur. Hash table implementers track this\ncollision likelihood by measuring the table’s load factor. It’s defined as\nthe number of entries divided by the number of buckets. So a hash table with\nfive entries and an array of 16 elements has a load factor of 0.3125. The higher\nthe load factor, the greater the chance of collisions.

\n

One way we mitigate collisions is by resizing the array. Just like the dynamic\narrays we implemented earlier, we reallocate and grow the hash table’s array as\nit fills up. Unlike a regular dynamic array, though, we won’t wait until the\narray is full. Instead, we pick a desired load factor and grow the array when\nit goes over that.

\n

20 . 2Collision Resolution

\n

Even with a very low load factor, collisions can still occur. The birthday\nparadox tells us that as the number of entries in the hash table\nincreases, the chance of collision increases very quickly. We can pick a large\narray size to reduce that, but it’s a losing game. Say we wanted to store a\nhundred items in a hash table. To keep the chance of collision below a\nstill-pretty-high 10%, we need an array with at least 47,015 elements. To get\nthe chance below 1% requires an array with 492,555 elements, over 4,000 empty\nbuckets for each one in use.

\n

A low load factor can make collisions rarer, but the\npigeonhole principle tells us we can never eliminate them entirely.\nIf you’ve got five pet pigeons and four holes to put them in, at least one hole\nis going to end up with more than one pigeon. With 18,446,744,073,709,551,616\ndifferent variable names, any reasonably sized array can potentially end up with\nmultiple keys in the same bucket.

\n

Thus we still have to handle collisions gracefully when they occur. Users don’t\nlike it when their programming language can look up variables correctly only\nmost of the time.

\n\n

20 . 2 . 1Separate chaining

\n

Techniques for resolving collisions fall into two broad categories. The first is\nseparate chaining. Instead of each bucket containing a single entry, we let\nit contain a collection of them. In the classic implementation, each bucket\npoints to a linked list of entries. To look up an entry, you find its bucket and\nthen walk the list until you find an entry with the matching key.

\"An\n

In catastrophically bad cases where every entry collides in the same bucket, the\ndata structure degrades into a single unsorted linked list with O(n) lookup.\nIn practice, it’s easy to avoid that by controlling the load factor and how\nentries get scattered across buckets. In typical separate-chained hash tables,\nit’s rare for a bucket to have more than one or two entries.

\n

Separate chaining is conceptually simple—it’s literally an array of linked\nlists. Most operations are straightforward to implement, even deletion which, as\nwe’ll see, can be a pain. But it’s not a great fit for modern CPUs. It has a lot\nof overhead from pointers and tends to scatter little linked list nodes around in memory which isn’t great for cache usage.

\n\n

20 . 2 . 2Open addressing

\n

The other technique is called open addressing or\n(confusingly) closed hashing. With this technique, all entries live directly\nin the bucket array, with one entry per bucket. If two entries collide in the\nsame bucket, we find a different empty bucket to use instead.

\n\n

Storing all entries in a single, big, contiguous array is great for keeping the\nmemory representation simple and fast. But it makes all of the operations on the\nhash table more complex. When inserting an entry, its bucket may be full,\nsending us to look at another bucket. That bucket itself may be occupied and so\non. This process of finding an available bucket is called probing, and the\norder that you examine buckets is a probe sequence.

\n

There are a number of algorithms for determining\nwhich buckets to probe and how to decide which entry goes in which bucket.\nThere’s been a ton of research here because even slight tweaks can have a large\nperformance impact. And, on a data structure as heavily used as hash tables,\nthat performance impact touches a very large number of real-world programs\nacross a range of hardware capabilities.

\n\n

As usual in this book, we’ll pick the simplest one that gets the job done\nefficiently. That’s good old linear probing. When looking for an entry, we\nlook in the first bucket its key maps to. If it’s not in there, we look in the\nvery next element in the array, and so on. If we reach the end, we wrap back\naround to the beginning.

\n

The good thing about linear probing is that it’s cache friendly. Since you walk\nthe array directly in memory order, it keeps the CPU’s cache lines full and\nhappy. The bad thing is that it’s prone to clustering. If you have a lot of\nentries with numerically similar key values, you can end up with a lot of\ncolliding, overflowing buckets right next to each other.

\n

Compared to separate chaining, open addressing can be harder to wrap your head\naround. I think of open addressing as similar to separate chaining except that\nthe “list” of nodes is threaded through the bucket array itself. Instead of\nstoring the links between them in pointers, the connections are calculated\nimplicitly by the order that you look through the buckets.

\n

The tricky part is that more than one of these implicit lists may be interleaved\ntogether. Let’s walk through an example that covers all the interesting cases.\nWe’ll ignore values for now and just worry about a set of keys. We start with an\nempty array of 8 buckets.

\"An\n

We decide to insert “bagel”. The first letter, “b” (ASCII value 98), modulo the\narray size (8) puts it in bucket 2.

\"Bagel\n

Next, we insert “jam”. That also wants to go in bucket 2 (106 mod 8 = 2), but\nthat bucket’s taken. We keep probing to the next bucket. It’s empty, so we put\nit there.

\"Jam\n

We insert “fruit”, which happily lands in bucket 6.

\"Fruit\n

Likewise, “migas” can go in its preferred bucket 5.

\"Migas\n

When we try to insert “eggs”, it also wants to be in bucket 5. That’s full, so we\nskip to 6. Bucket 6 is also full. Note that the entry in there is not part of\nthe same probe sequence. “Fruit” is in its preferred bucket, 6. So the 5 and 6\nsequences have collided and are interleaved. We skip over that and finally put\n“eggs” in bucket 7.

\"Eggs\n

We run into a similar problem with “nuts”. It can’t land in 6 like it wants to.\nNor can it go into 7. So we keep going. But we’ve reached the end of the array,\nso we wrap back around to 0 and put it there.

\"Nuts\n

In practice, the interleaving turns out to not be much of a problem. Even in\nseparate chaining, we need to walk the list to check each entry’s key because\nmultiple keys can reduce to the same bucket. With open addressing, we need to do\nthat same check, and that also covers the case where you are stepping over\nentries that “belong” to a different original bucket.

\n

20 . 3Hash Functions

\n

We can now build ourselves a reasonably efficient table for storing variable\nnames up to eight characters long, but that limitation is still annoying. In\norder to relax the last constraint, we need a way to take a string of any length\nand convert it to a fixed-size integer.

\n

Finally, we get to the “hash” part of “hash table”. A hash function takes\nsome larger blob of data and “hashes” it to produce a fixed-size integer hash\ncode whose value depends on all of the bits of the original data. A good hash function has three main goals:

\n\n
    \n
  • \n

    It must be deterministic. The same input must always hash to the same\nnumber. If the same variable ends up in different buckets at different\npoints in time, it’s gonna get really hard to find it.

    \n
    • \n
  • \n

    It must be uniform. Given a typical set of inputs, it should produce a\nwide and evenly distributed range of output numbers, with as few clumps or\npatterns as possible. We want it to scatter\nvalues across the whole numeric range to minimize collisions and clustering.

    \n
    • \n
  • \n

    It must be fast. Every operation on the hash table requires us to hash\nthe key first. If hashing is slow, it can potentially cancel out the speed\nof the underlying array storage.

    \n
    • \n
\n\n

There is a veritable pile of hash functions out there. Some are old and\noptimized for architectures no one uses anymore. Some are designed to be fast,\nothers cryptographically secure. Some take advantage of vector instructions and\ncache sizes for specific chips, others aim to maximize portability.

\n

There are people out there for whom designing and evaluating hash functions is,\nlike, their jam. I admire them, but I’m not mathematically astute enough to\nbe one. So for clox, I picked a simple, well-worn hash function called\nFNV-1a that’s served me fine over the years. Consider trying out different ones in your code and see if they make\na difference.

\n\n

OK, that’s a quick run through of buckets, load factors, open addressing,\ncollision resolution, and hash functions. That’s an awful lot of text and not a\nlot of real code. Don’t worry if it still seems vague. Once we’re done coding it\nup, it will all click into place.

\n

20 . 4Building a Hash Table

\n

The great thing about hash tables compared to other classic techniques like\nbalanced search trees is that the actual data structure is so simple. Ours goes\ninto a new module.

\n
table.h
\ncreate new file
\n
#ifndef clox_table_h\n#define clox_table_h\n\n#include "common.h"\n#include "value.h"\n\ntypedef struct {\n  int count;\n  int capacity;\n  Entry* entries;\n} Table;\n\n#endif\n
\n
table.h, create new file
\n\n

A hash table is an array of entries. As in our dynamic array earlier, we keep\ntrack of both the allocated size of the array (capacity) and the number of\nkey/value pairs currently stored in it (count). The ratio of count to capacity\nis exactly the load factor of the hash table.

\n

Each entry is one of these:

\n
#include "value.h"\n
table.h
\n
\n\ntypedef struct {\n  ObjString* key;\n  Value value;\n} Entry;\n
\n\ntypedef struct {\n
\n
table.h
\n\n

It’s a simple key/value pair. Since the key is always a string, we store the ObjString pointer directly instead of\nwrapping it in a Value. It’s a little faster and smaller this way.

\n\n

To create a new, empty hash table, we declare a constructor-like function.

\n
} Table;\n\n
table.h
\nadd after struct Table
\n
void initTable(Table* table);\n\n
#endif\n
\n
table.h, add after struct Table
\n\n

We need a new implementation file to define that. While we’re at it, let’s get\nall of the pesky includes out of the way.

\n
table.c
\ncreate new file
\n
#include <stdlib.h>\n#include <string.h>\n\n#include "memory.h"\n#include "object.h"\n#include "table.h"\n#include "value.h"\n\nvoid initTable(Table* table) {\n  table->count = 0;\n  table->capacity = 0;\n  table->entries = NULL;\n}\n
\n
table.c, create new file
\n\n

As in our dynamic value array type, a hash table initially starts with zero\ncapacity and a NULL array. We don’t allocate anything until needed. Assuming\nwe do eventually allocate something, we need to be able to free it too.

\n
void initTable(Table* table);\n
table.h
\nadd after initTable()
\n
void freeTable(Table* table);\n
\n\n#endif\n
\n
table.h, add after initTable()
\n\n

And its glorious implementation:

\n
table.c
\nadd after initTable()
\n
void freeTable(Table* table) {\n  FREE_ARRAY(Entry, table->entries, table->capacity);\n  initTable(table);\n}\n
\n
table.c, add after initTable()
\n\n

Again, it looks just like a dynamic array. In fact, you can think of a hash\ntable as basically a dynamic array with a really strange policy for inserting\nitems. We don’t need to check for NULL here since FREE_ARRAY() already\nhandles that gracefully.

\n

20 . 4 . 1Hashing strings

\n

Before we can start putting entries in the table, we need to, well, hash them.\nTo ensure that the entries get distributed uniformly throughout the array, we\nwant a good hash function that looks at all of the bits of the key string. If it\nlooked at, say, only the first few characters, then a series of strings that all\nshared the same prefix would end up colliding in the same bucket.

\n

On the other hand, walking the entire string to calculate the hash is kind of\nslow. We’d lose some of the performance benefit of the hash table if we had to\nwalk the string every time we looked for a key in the table. So we’ll do the\nobvious thing: cache it.

\n

Over in the “object” module in ObjString, we add:

\n
  char* chars;\n
object.h
\nin struct ObjString
\n
  uint32_t hash;\n
};\n
\n
object.h, in struct ObjString
\n\n

Each ObjString stores the hash code for its string. Since strings are immutable\nin Lox, we can calculate the hash code once up front and be certain that it will\nnever get invalidated. Caching it eagerly makes a kind of sense: allocating the\nstring and copying its characters over is already an O(n) operation, so it’s a\ngood time to also do the O(n) calculation of the string’s hash.

\n

Whenever we call the internal function to allocate a string, we pass in its\nhash code.

\n
object.c
\nfunction allocateString()
\nreplace 1 line
\n
static ObjString* allocateString(char* chars, int length,\n                                 uint32_t hash) {\n
  ObjString* string = ALLOCATE_OBJ(ObjString, OBJ_STRING);\n
\n
object.c, function allocateString(), replace 1 line
\n\n

That function simply stores the hash in the struct.

\n
  string->chars = chars;\n
object.c
\nin allocateString()
\n
  string->hash = hash;\n
  return string;\n}\n
\n
object.c, in allocateString()
\n\n

The fun happens over at the callers. allocateString() is called from two\nplaces: the function that copies a string and the one that takes ownership of an\nexisting dynamically allocated string. We’ll start with the first.

\n
ObjString* copyString(const char* chars, int length) {\n
object.c
\nin copyString()
\n
  uint32_t hash = hashString(chars, length);\n
  char* heapChars = ALLOCATE(char, length + 1);\n
\n
object.c, in copyString()
\n\n

No magic here. We calculate the hash code and then pass it along.

\n
  memcpy(heapChars, chars, length);\n  heapChars[length] = '\\0';\n
object.c
\nin copyString()
\nreplace 1 line
\n
  return allocateString(heapChars, length, hash);\n
}\n
\n
object.c, in copyString(), replace 1 line
\n\n

The other string function is similar.

\n
ObjString* takeString(char* chars, int length) {\n
object.c
\nin takeString()
\nreplace 1 line
\n
  uint32_t hash = hashString(chars, length);\n  return allocateString(chars, length, hash);\n
}\n
\n
object.c, in takeString(), replace 1 line
\n\n

The interesting code is over here:

\n
object.c
\nadd after allocateString()
\n
static uint32_t hashString(const char* key, int length) {\n  uint32_t hash = 2166136261u;\n  for (int i = 0; i < length; i++) {\n    hash ^= (uint8_t)key[i];\n    hash *= 16777619;\n  }\n  return hash;\n}\n
\n
object.c, add after allocateString()
\n\n

This is the actual bona fide “hash function” in clox. The algorithm is called\n“FNV-1a”, and is the shortest decent hash function I know. Brevity is certainly\na virtue in a book that aims to show you every line of code.

\n

The basic idea is pretty simple, and many hash functions follow the same\npattern. You start with some initial hash value, usually a constant with certain\ncarefully chosen mathematical properties. Then you walk the data to be hashed.\nFor each byte (or sometimes word), you mix the bits into the hash value somehow,\nand then scramble the resulting bits around some.

\n

What it means to “mix” and “scramble” can get pretty sophisticated. Ultimately,\nthough, the basic goal is uniformity—we want the resulting hash values to\nbe as widely scattered around the numeric range as possible to avoid collisions\nand clustering.

\n

20 . 4 . 2Inserting entries

\n

Now that string objects know their hash code, we can start putting them into\nhash tables.

\n
void freeTable(Table* table);\n
table.h
\nadd after freeTable()
\n
bool tableSet(Table* table, ObjString* key, Value value);\n
\n\n#endif\n
\n
table.h, add after freeTable()
\n\n

This function adds the given key/value pair to the given hash table. If an entry\nfor that key is already present, the new value overwrites the old value. The\nfunction returns true if a new entry was added. Here’s the implementation:

\n
table.c
\nadd after freeTable()
\n
bool tableSet(Table* table, ObjString* key, Value value) {\n  Entry* entry = findEntry(table->entries, table->capacity, key);\n  bool isNewKey = entry->key == NULL;\n  if (isNewKey) table->count++;\n\n  entry->key = key;\n  entry->value = value;\n  return isNewKey;\n}\n
\n
table.c, add after freeTable()
\n\n

Most of the interesting logic is in findEntry() which we’ll get to soon. That\nfunction’s job is to take a key and figure out which bucket in the array it\nshould go in. It returns a pointer to that bucket—the address of the Entry in\nthe array.

\n

Once we have a bucket, inserting is straightforward. We update the hash table’s\nsize, taking care to not increase the count if we overwrote the value for an\nalready-present key. Then we copy the key and value into the corresponding\nfields in the Entry.

\n

We’re missing a little something here, though. We haven’t actually allocated the\nEntry array yet. Oops! Before we can insert anything, we need to make sure we\nhave an array, and that it’s big enough.

\n
bool tableSet(Table* table, ObjString* key, Value value) {\n
table.c
\nin tableSet()
\n
  if (table->count + 1 > table->capacity * TABLE_MAX_LOAD) {\n    int capacity = GROW_CAPACITY(table->capacity);\n    adjustCapacity(table, capacity);\n  }\n\n
  Entry* entry = findEntry(table->entries, table->capacity, key);\n
\n
table.c, in tableSet()
\n\n

This is similar to the code we wrote a while back for growing a dynamic array.\nIf we don’t have enough capacity to insert an item, we reallocate and grow the\narray. The GROW_CAPACITY() macro takes an existing capacity and grows it by\na multiple to ensure that we get amortized constant performance over a series\nof inserts.

\n

The interesting difference here is that TABLE_MAX_LOAD constant.

\n
#include "value.h"\n\n
table.c
\n
#define TABLE_MAX_LOAD 0.75\n\n
void initTable(Table* table) {\n
\n
table.c
\n\n

This is how we manage the table’s load factor. We don’t\ngrow when the capacity is completely full. Instead, we grow the array before\nthen, when the array becomes at least 75% full.

\n\n

We’ll get to the implementation of adjustCapacity() soon. First, let’s look\nat that findEntry() function you’ve been wondering about.

\n
table.c
\nadd after freeTable()
\n
static Entry* findEntry(Entry* entries, int capacity,\n                        ObjString* key) {\n  uint32_t index = key->hash % capacity;\n  for (;;) {\n    Entry* entry = &entries[index];\n    if (entry->key == key || entry->key == NULL) {\n      return entry;\n    }\n\n    index = (index + 1) % capacity;\n  }\n}\n
\n
table.c, add after freeTable()
\n\n

This function is the real core of the hash table. It’s responsible for taking a\nkey and an array of buckets, and figuring out which bucket the entry belongs in.\nThis function is also where linear probing and collision handling come into\nplay. We’ll use findEntry() both to look up existing entries in the hash\ntable and to decide where to insert new ones.

\n

For all that, there isn’t much to it. First, we use modulo to map the key’s hash\ncode to an index within the array’s bounds. That gives us a bucket index where,\nideally, we’ll be able to find or place the entry.

\n

There are a few cases to check for:

\n
    \n
  • \n

    If the key for the Entry at that array index is NULL, then the bucket is\nempty. If we’re using findEntry() to look up something in the hash table,\nthis means it isn’t there. If we’re using it to insert, it means we’ve found\na place to add the new entry.

    \n
    • \n
  • \n

    If the key in the bucket is equal to the key we’re\nlooking for, then that key is already present in the table. If we’re doing a\nlookup, that’s good—we’ve found the key we seek. If we’re doing an insert,\nthis means we’ll be replacing the value for that key instead of adding a new\nentry.

    \n
    • \n
\n\n
    \n
  • Otherwise, the bucket has an entry in it, but with a different key. This is\na collision. In that case, we start probing. That’s what that for loop\ndoes. We start at the bucket where the entry would ideally go. If that\nbucket is empty or has the same key, we’re done. Otherwise, we advance to\nthe next element—this is the linear part of “linear probing”—and\ncheck there. If we go past the end of the array, that second modulo operator\nwraps us back around to the beginning.
    • \n
\n

We exit the loop when we find either an empty bucket or a bucket with the same\nkey as the one we’re looking for. You might be wondering about an infinite loop.\nWhat if we collide with every bucket? Fortunately, that can’t happen thanks to\nour load factor. Because we grow the array as soon as it gets close to being\nfull, we know there will always be empty buckets.

\n

We return directly from within the loop, yielding a pointer to the found Entry\nso the caller can either insert something into it or read from it. Way back in\ntableSet(), the function that first kicked this off, we store the new entry in\nthat returned bucket and we’re done.

\n

20 . 4 . 3Allocating and resizing

\n

Before we can put entries in the hash table, we do need a place to actually\nstore them. We need to allocate an array of buckets. That happens in this\nfunction:

\n
table.c
\nadd after findEntry()
\n
static void adjustCapacity(Table* table, int capacity) {\n  Entry* entries = ALLOCATE(Entry, capacity);\n  for (int i = 0; i < capacity; i++) {\n    entries[i].key = NULL;\n    entries[i].value = NIL_VAL;\n  }\n\n  table->entries = entries;\n  table->capacity = capacity;\n}\n
\n
table.c, add after findEntry()
\n\n

We create a bucket array with capacity entries. After we allocate the array,\nwe initialize every element to be an empty bucket and then store the array (and\nits capacity) in the hash table’s main struct. This code is fine for when we\ninsert the very first entry into the table, and we require the first allocation\nof the array. But what about when we already have one and we need to grow it?

\n

Back when we were doing a dynamic array, we could just use realloc() and let\nthe C standard library copy everything over. That doesn’t work for a hash table.\nRemember that to choose the bucket for each entry, we take its hash key modulo\nthe array size. That means that when the array size changes, entries may end up\nin different buckets.

\n

Those new buckets may have new collisions that we need to deal with. So the\nsimplest way to get every entry where it belongs is to rebuild the table from\nscratch by re-inserting every entry into the new empty array.

\n
    entries[i].value = NIL_VAL;\n  }\n
table.c
\nin adjustCapacity()
\n
\n\n  for (int i = 0; i < table->capacity; i++) {\n    Entry* entry = &table->entries[i];\n    if (entry->key == NULL) continue;\n\n    Entry* dest = findEntry(entries, capacity, entry->key);\n    dest->key = entry->key;\n    dest->value = entry->value;\n  }\n
\n\n  table->entries = entries;\n
\n
table.c, in adjustCapacity()
\n\n

We walk through the old array front to back. Any time we find a non-empty\nbucket, we insert that entry into the new array. We use findEntry(), passing\nin the new array instead of the one currently stored in the Table. (This is\nwhy findEntry() takes a pointer directly to an Entry array and not the whole\nTable struct. That way, we can pass the new array and capacity before we’ve\nstored those in the struct.)

\n

After that’s done, we can release the memory for the old array.

\n
    dest->value = entry->value;\n  }\n\n
table.c
\nin adjustCapacity()
\n
  FREE_ARRAY(Entry, table->entries, table->capacity);\n
  table->entries = entries;\n
\n
table.c, in adjustCapacity()
\n\n

With that, we have a hash table that we can stuff as many entries into as we\nlike. It handles overwriting existing keys and growing itself as needed to\nmaintain the desired load capacity.

\n

While we’re at it, let’s also define a helper function for copying all of the\nentries of one hash table into another.

\n
bool tableSet(Table* table, ObjString* key, Value value);\n
table.h
\nadd after tableSet()
\n
void tableAddAll(Table* from, Table* to);\n
\n\n#endif\n
\n
table.h, add after tableSet()
\n\n

We won’t need this until much later when we support method inheritance, but we\nmay as well implement it now while we’ve got all the hash table stuff fresh in\nour minds.

\n
table.c
\nadd after tableSet()
\n
void tableAddAll(Table* from, Table* to) {\n  for (int i = 0; i < from->capacity; i++) {\n    Entry* entry = &from->entries[i];\n    if (entry->key != NULL) {\n      tableSet(to, entry->key, entry->value);\n    }\n  }\n}\n
\n
table.c, add after tableSet()
\n\n

There’s not much to say about this. It walks the bucket array of the source hash\ntable. Whenever it finds a non-empty bucket, it adds the entry to the\ndestination hash table using the tableSet() function we recently defined.

\n

20 . 4 . 4Retrieving values

\n

Now that our hash table contains some stuff, let’s start pulling things back\nout. Given a key, we can look up the corresponding value, if there is one, with\nthis function:

\n
void freeTable(Table* table);\n
table.h
\nadd after freeTable()
\n
bool tableGet(Table* table, ObjString* key, Value* value);\n
bool tableSet(Table* table, ObjString* key, Value value);\n
\n
table.h, add after freeTable()
\n\n

You pass in a table and a key. If it finds an entry with that key, it returns\ntrue, otherwise it returns false. If the entry exists, the value output\nparameter points to the resulting value.

\n

Since findEntry() already does the hard work, the implementation isn’t bad.

\n
table.c
\nadd after findEntry()
\n
bool tableGet(Table* table, ObjString* key, Value* value) {\n  if (table->count == 0) return false;\n\n  Entry* entry = findEntry(table->entries, table->capacity, key);\n  if (entry->key == NULL) return false;\n\n  *value = entry->value;\n  return true;\n}\n
\n
table.c, add after findEntry()
\n\n

If the table is completely empty, we definitely won’t find the entry, so we\ncheck for that first. This isn’t just an optimization—it also ensures that we\ndon’t try to access the bucket array when the array is NULL. Otherwise, we let\nfindEntry() work its magic. That returns a pointer to a bucket. If the bucket\nis empty, which we detect by seeing if the key is NULL, then we didn’t find an\nEntry with our key. If findEntry() does return a non-empty Entry, then that’s\nour match. We take the Entry’s value and copy it to the output parameter so the\ncaller can get it. Piece of cake.

\n

20 . 4 . 5Deleting entries

\n

There is one more fundamental operation a full-featured hash table needs to\nsupport: removing an entry. This seems pretty obvious, if you can add things,\nyou should be able to un-add them, right? But you’d be surprised how many\ntutorials on hash tables omit this.

\n

I could have taken that route too. In fact, we use deletion in clox only in a\ntiny edge case in the VM. But if you want to actually understand how to\ncompletely implement a hash table, this feels important. I can sympathize with\ntheir desire to overlook it. As we’ll see, deleting from a hash table that uses\nopen addressing is tricky.

\n\n

At least the declaration is simple.

\n
bool tableSet(Table* table, ObjString* key, Value value);\n
table.h
\nadd after tableSet()
\n
bool tableDelete(Table* table, ObjString* key);\n
void tableAddAll(Table* from, Table* to);\n
\n
table.h, add after tableSet()
\n\n

The obvious approach is to mirror insertion. Use findEntry() to look up the\nentry’s bucket. Then clear out the bucket. Done!

\n

In cases where there are no collisions, that works fine. But if a collision has\noccurred, then the bucket where the entry lives may be part of one or more\nimplicit probe sequences. For example, here’s a hash table containing three keys\nall with the same preferred bucket, 2:

\"A\n

Remember that when we’re walking a probe sequence to find an entry, we know\nwe’ve reached the end of a sequence and that the entry isn’t present when we hit\nan empty bucket. It’s like the probe sequence is a list of entries and an empty\nentry terminates that list.

\n

If we delete “biscuit” by simply clearing the Entry, then we break that probe\nsequence in the middle, leaving the trailing entries orphaned and unreachable.\nSort of like removing a node from a linked list without relinking the pointer\nfrom the previous node to the next one.

\n

If we later try to look for “jam”, we’d start at “bagel”, stop at the next\nempty Entry, and never find it.

\"The\n

To solve this, most implementations use a trick called tombstones. Instead of clearing the entry on\ndeletion, we replace it with a special sentinel entry called a “tombstone”. When\nwe are following a probe sequence during a lookup, and we hit a tombstone, we\ndon’t treat it like an empty slot and stop iterating. Instead, we keep going\nso that deleting an entry doesn’t break any implicit collision chains and we can\nstill find entries after it.

\"Instead\n

The code looks like this:

\n
table.c
\nadd after tableSet()
\n
bool tableDelete(Table* table, ObjString* key) {\n  if (table->count == 0) return false;\n\n  // Find the entry.\n  Entry* entry = findEntry(table->entries, table->capacity, key);\n  if (entry->key == NULL) return false;\n\n  // Place a tombstone in the entry.\n  entry->key = NULL;\n  entry->value = BOOL_VAL(true);\n  return true;\n}\n
\n
table.c, add after tableSet()
\n\n

First, we find the bucket containing the entry we want to delete. (If we don’t\nfind it, there’s nothing to delete, so we bail out.) We replace the entry with a\ntombstone. In clox, we use a NULL key and a true value to represent that,\nbut any representation that can’t be confused with an empty bucket or a valid\nentry works.

\n\n

That’s all we need to do to delete an entry. Simple and fast. But all of the\nother operations need to correctly handle tombstones too. A tombstone is a sort\nof “half” entry. It has some of the characteristics of a present entry, and some\nof the characteristics of an empty one.

\n

When we are following a probe sequence during a lookup, and we hit a tombstone,\nwe note it and keep going.

\n
  for (;;) {\n    Entry* entry = &entries[index];\n
table.c
\nin findEntry()
\nreplace 3 lines
\n
    if (entry->key == NULL) {\n      if (IS_NIL(entry->value)) {\n        // Empty entry.\n        return tombstone != NULL ? tombstone : entry;\n      } else {\n        // We found a tombstone.\n        if (tombstone == NULL) tombstone = entry;\n      }\n    } else if (entry->key == key) {\n      // We found the key.\n      return entry;\n    }\n
\n\n    index = (index + 1) % capacity;\n
\n
table.c, in findEntry(), replace 3 lines
\n\n

The first time we pass a tombstone, we store it in this local variable:

\n
  uint32_t index = key->hash % capacity;\n
table.c
\nin findEntry()
\n
  Entry* tombstone = NULL;\n\n
  for (;;) {\n
\n
table.c, in findEntry()
\n\n

If we reach a truly empty entry, then the key isn’t present. In that case, if we\nhave passed a tombstone, we return its bucket instead of the later empty one. If\nwe’re calling findEntry() in order to insert a node, that lets us treat the\ntombstone bucket as empty and reuse it for the new entry.

\n

Reusing tombstone slots automatically like this helps reduce the number of\ntombstones wasting space in the bucket array. In typical use cases where there\nis a mixture of insertions and deletions, the number of tombstones grows for a\nwhile and then tends to stabilize.

\n

Even so, there’s no guarantee that a large number of deletes won’t cause the\narray to be full of tombstones. In the very worst case, we could end up with\nno empty buckets. That would be bad because, remember, the only thing\npreventing an infinite loop in findEntry() is the assumption that we’ll\neventually hit an empty bucket.

\n

So we need to be thoughtful about how tombstones interact with the table’s load\nfactor and resizing. The key question is, when calculating the load factor,\nshould we treat tombstones like full buckets or empty ones?

\n

20 . 4 . 6Counting tombstones

\n

If we treat tombstones like full buckets, then we may end up with a bigger array\nthan we probably need because it artificially inflates the load factor. There\nare tombstones we could reuse, but they aren’t treated as unused so we end up\ngrowing the array prematurely.

\n

But if we treat tombstones like empty buckets and don’t include them in the\nload factor, then we run the risk of ending up with no actual empty buckets to\nterminate a lookup. An infinite loop is a much worse problem than a few extra\narray slots, so for load factor, we consider tombstones to be full buckets.

\n

That’s why we don’t reduce the count when deleting an entry in the previous\ncode. The count is no longer the number of entries in the hash table, it’s the\nnumber of entries plus tombstones. That implies that we increment the count\nduring insertion only if the new entry goes into an entirely empty bucket.

\n
  bool isNewKey = entry->key == NULL;\n
table.c
\nin tableSet()
\nreplace 1 line
\n
  if (isNewKey && IS_NIL(entry->value)) table->count++;\n
\n\n  entry->key = key;\n
\n
table.c, in tableSet(), replace 1 line
\n\n

If we are replacing a tombstone with a new entry, the bucket has already been\naccounted for and the count doesn’t change.

\n

When we resize the array, we allocate a new array and re-insert all of the\nexisting entries into it. During that process, we don’t copy the tombstones\nover. They don’t add any value since we’re rebuilding the probe sequences\nanyway, and would just slow down lookups. That means we need to recalculate the\ncount since it may change during a resize. So we clear it out:

\n
  }\n\n
table.c
\nin adjustCapacity()
\n
  table->count = 0;\n
  for (int i = 0; i < table->capacity; i++) {\n
\n
table.c, in adjustCapacity()
\n\n

Then each time we find a non-tombstone entry, we increment it.

\n
    dest->value = entry->value;\n
table.c
\nin adjustCapacity()
\n
    table->count++;\n
  }\n
\n
table.c, in adjustCapacity()
\n\n

This means that when we grow the capacity, we may end up with fewer entries in\nthe resulting larger array because all of the tombstones get discarded. That’s a\nlittle wasteful, but not a huge practical problem.

\n

I find it interesting that much of the work to support deleting entries is in\nfindEntry() and adjustCapacity(). The actual delete logic is quite simple\nand fast. In practice, deletions tend to be rare, so you’d expect a hash table\nto do as much work as it can in the delete function and leave the other\nfunctions alone to keep them faster. With our tombstone approach, deletes are\nfast, but lookups get penalized.

\n

I did a little benchmarking to test this out in a few different deletion\nscenarios. I was surprised to discover that tombstones did end up being faster\noverall compared to doing all the work during deletion to reinsert the affected\nentries.

\n

But if you think about it, it’s not that the tombstone approach pushes the work\nof fully deleting an entry to other operations, it’s more that it makes deleting\nlazy. At first, it does the minimal work to turn the entry into a tombstone.\nThat can cause a penalty when later lookups have to skip over it. But it also\nallows that tombstone bucket to be reused by a later insert too. That reuse is a\nvery efficient way to avoid the cost of rearranging all of the following\naffected entries. You basically recycle a node in the chain of probed entries.\nIt’s a neat trick.

\n

20 . 5String Interning

\n

We’ve got ourselves a hash table that mostly works, though it has a critical\nflaw in its center. Also, we aren’t using it for anything yet. It’s time to\naddress both of those and, in the process, learn a classic technique used by\ninterpreters.

\n

The reason the hash table doesn’t totally work is that when findEntry() checks\nto see if an existing key matches the one it’s looking for, it uses == to\ncompare two strings for equality. That only returns true if the two keys are the\nexact same string in memory. Two separate strings with the same characters\nshould be considered equal, but aren’t.

\n

Remember, back when we added strings in the last chapter, we added explicit\nsupport to compare the strings character-by-character in order to get\ntrue value equality. We could do that in findEntry(), but that’s slow.

\n\n

Instead, we’ll use a technique called string interning. The core problem is\nthat it’s possible to have different strings in memory with the same characters.\nThose need to behave like equivalent values even though they are distinct\nobjects. They’re essentially duplicates, and we have to compare all of their\nbytes to detect that.

\n

String interning is a process of deduplication. We\ncreate a collection of “interned” strings. Any string in that collection is\nguaranteed to be textually distinct from all others. When you intern a string,\nyou look for a matching string in the collection. If found, you use that\noriginal one. Otherwise, the string you have is unique, so you add it to the\ncollection.

\n\n

In this way, you know that each sequence of characters is represented by only\none string in memory. This makes value equality trivial. If two strings point\nto the same address in memory, they are obviously the same string and must be\nequal. And, because we know strings are unique, if two strings point to\ndifferent addresses, they must be distinct strings.

\n

Thus, pointer equality exactly matches value equality. Which in turn means that\nour existing == in findEntry() does the right thing. Or, at least, it will\nonce we intern all the strings. In order to reliably deduplicate all strings,\nthe VM needs to be able to find every string that’s created. We do that by\ngiving it a hash table to store them all.

\n
  Value* stackTop;\n
vm.h
\nin struct VM
\n
  Table strings;\n
  Obj* objects;\n
\n
vm.h, in struct VM
\n\n

As usual, we need an include.

\n
#include "chunk.h"\n
vm.h
\n
#include "table.h"\n
#include "value.h"\n
\n
vm.h
\n\n

When we spin up a new VM, the string table is empty.

\n
  vm.objects = NULL;\n
vm.c
\nin initVM()
\n
  initTable(&vm.strings);\n
}\n
\n
vm.c, in initVM()
\n\n

And when we shut down the VM, we clean up any resources used by the table.

\n
void freeVM() {\n
vm.c
\nin freeVM()
\n
  freeTable(&vm.strings);\n
  freeObjects();\n
\n
vm.c, in freeVM()
\n\n

Some languages have a separate type or an explicit step to intern a string. For\nclox, we’ll automatically intern every one. That means whenever we create a new\nunique string, we add it to the table.

\n
  string->hash = hash;\n
object.c
\nin allocateString()
\n
  tableSet(&vm.strings, string, NIL_VAL);\n
  return string;\n
\n
object.c, in allocateString()
\n\n

We’re using the table more like a hash set than a hash table. The keys are\nthe strings and those are all we care about, so we just use nil for the\nvalues.

\n

This gets a string into the table assuming that it’s unique, but we need to\nactually check for duplication before we get here. We do that in the two\nhigher-level functions that call allocateString(). Here’s one:

\n
  uint32_t hash = hashString(chars, length);\n
object.c
\nin copyString()
\n
  ObjString* interned = tableFindString(&vm.strings, chars, length,\n                                        hash);\n  if (interned != NULL) return interned;\n\n
  char* heapChars = ALLOCATE(char, length + 1);\n
\n
object.c, in copyString()
\n\n

When copying a string into a new LoxString, we look it up in the string table\nfirst. If we find it, instead of “copying”, we just return a reference to that\nstring. Otherwise, we fall through, allocate a new string, and store it in the\nstring table.

\n

Taking ownership of a string is a little different.

\n
  uint32_t hash = hashString(chars, length);\n
object.c
\nin takeString()
\n
  ObjString* interned = tableFindString(&vm.strings, chars, length,\n                                        hash);\n  if (interned != NULL) {\n    FREE_ARRAY(char, chars, length + 1);\n    return interned;\n  }\n\n
  return allocateString(chars, length, hash);\n
\n
object.c, in takeString()
\n\n

Again, we look up the string in the string table first. If we find it, before we\nreturn it, we free the memory for the string that was passed in. Since ownership\nis being passed to this function and we no longer need the duplicate string,\nit’s up to us to free it.

\n

Before we get to the new function we need to write, there’s one more include.

\n
#include "object.h"\n
object.c
\n
#include "table.h"\n
#include "value.h"\n
\n
object.c
\n\n

To look for a string in the table, we can’t use the normal tableGet() function\nbecause that calls findEntry(), which has the exact problem with duplicate\nstrings that we’re trying to fix right now. Instead, we use this new function:

\n
void tableAddAll(Table* from, Table* to);\n
table.h
\nadd after tableAddAll()
\n
ObjString* tableFindString(Table* table, const char* chars,\n                           int length, uint32_t hash);\n
\n\n#endif\n
\n
table.h, add after tableAddAll()
\n\n

The implementation looks like so:

\n
table.c
\nadd after tableAddAll()
\n
ObjString* tableFindString(Table* table, const char* chars,\n                           int length, uint32_t hash) {\n  if (table->count == 0) return NULL;\n\n  uint32_t index = hash % table->capacity;\n  for (;;) {\n    Entry* entry = &table->entries[index];\n    if (entry->key == NULL) {\n      // Stop if we find an empty non-tombstone entry.\n      if (IS_NIL(entry->value)) return NULL;\n    } else if (entry->key->length == length &&\n        entry->key->hash == hash &&\n        memcmp(entry->key->chars, chars, length) == 0) {\n      // We found it.\n      return entry->key;\n    }\n\n    index = (index + 1) % table->capacity;\n  }\n}\n
\n
table.c, add after tableAddAll()
\n\n

It appears we have copy-pasted findEntry(). There is a lot of redundancy, but\nalso a couple of key differences. First, we pass in the raw character array of\nthe key we’re looking for instead of an ObjString. At the point that we call\nthis, we haven’t created an ObjString yet.

\n

Second, when checking to see if we found the key, we look at the actual strings.\nWe first see if they have matching lengths and hashes. Those are quick to check\nand if they aren’t equal, the strings definitely aren’t the same.

\n

If there is a hash collision, we do an actual character-by-character string\ncomparison. This is the one place in the VM where we actually test strings for\ntextual equality. We do it here to deduplicate strings and then the rest of the\nVM can take for granted that any two strings at different addresses in memory\nmust have different contents.

\n

In fact, now that we’ve interned all the strings, we can take advantage of it in\nthe bytecode interpreter. When a user does == on two objects that happen to be\nstrings, we don’t need to test the characters any more.

\n
    case VAL_NUMBER: return AS_NUMBER(a) == AS_NUMBER(b);\n
value.c
\nin valuesEqual()
\nreplace 7 lines
\n
    case VAL_OBJ:    return AS_OBJ(a) == AS_OBJ(b);\n
    default:         return false; // Unreachable.\n
\n
value.c, in valuesEqual(), replace 7 lines
\n\n

We’ve added a little overhead when creating strings to intern them. But in\nreturn, at runtime, the equality operator on strings is much faster. With that,\nwe have a full-featured hash table ready for us to use for tracking variables,\ninstances, or any other key-value pairs that might show up.

\n

We also sped up testing strings for equality. This is nice for when the user\ndoes == on strings. But it’s even more critical in a dynamically typed\nlanguage like Lox where method calls and instance fields are looked up by name\nat runtime. If testing a string for equality is slow, then that means looking up\na method by name is slow. And if that’s slow in your object-oriented language,\nthen everything is slow.

\n
\n

Challenges

\n
    \n
  1. \n

    In clox, we happen to only need keys that are strings, so the hash table we\nbuilt is hardcoded for that key type. If we exposed hash tables to Lox users\nas a first-class collection, it would be useful to support different kinds\nof keys.

    \n

    Add support for keys of the other primitive types: numbers, Booleans, and\nnil. Later, clox will support user-defined classes. If we want to support\nkeys that are instances of those classes, what kind of complexity does that\nadd?

    \n
    2. \n
  2. \n

    Hash tables have a lot of knobs you can tweak that affect their performance.\nYou decide whether to use separate chaining or open addressing. Depending on\nwhich fork in that road you take, you can tune how many entries are stored\nin each node, or the probing strategy you use. You control the hash\nfunction, load factor, and growth rate.

    \n

    All of this variety wasn’t created just to give CS doctoral candidates\nsomething to publish theses on: each has its\nuses in the many varied domains and hardware scenarios where hashing comes\ninto play. Look up a few hash table implementations in different open source\nsystems, research the choices they made, and try to figure out why they did\nthings that way.

    \n
    3. \n
  3. \n

    Benchmarking a hash table is notoriously difficult. A hash table\nimplementation may perform well with some keysets and poorly with others. It\nmay work well at small sizes but degrade as it grows, or vice versa. It may\nchoke when deletions are common, but fly when they aren’t. Creating\nbenchmarks that accurately represent how your users will use the hash table\nis a challenge.

    \n

    Write a handful of different benchmark programs to validate our hash table\nimplementation. How does the performance vary between them? Why did you\nchoose the specific test cases you chose?

    \n
    4. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/index.css", "content": "@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-roman.woff\") format(\"woff\");\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-italic.woff\") format(\"woff\");\n font-style: italic;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-semibold.woff\") format(\"woff\");\n font-weight: 600;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-semibolditalic.woff\") format(\"woff\");\n font-style: italic;\n font-weight: 600;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-bold.woff\") format(\"woff\");\n font-weight: bold;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-bolditalic.woff\") format(\"woff\");\n font-style: italic;\n font-weight: bold;\n}\nbody, h1, h2, h3, h4, p, blockquote, code, ul, ol, dl, dd, img {\n margin: 0;\n}\n\nimg {\n outline: none;\n}\n\nimg.arrow {\n width: auto;\n height: 11px;\n}\n\nimg.dot {\n width: auto;\n height: 18px;\n vertical-align: text-bottom;\n}\n\nbody {\n color: #222;\n font: normal 16px/24px \"Crimson\", Georgia, serif;\n}\n\n.sign-up {\n padding: 12px;\n margin: 24px 0 24px 0;\n background: #fcf6e8;\n color: #bf9540;\n border-radius: 3px;\n}\n.sign-up form {\n display: flex;\n}\n.sign-up input {\n padding: 4px;\n font: 16px \"Source Sans Pro\", sans-serif;\n outline: none;\n border-radius: 3px;\n border: solid 2px #ffd580;\n color: #825e17;\n height: 32px;\n}\n.sign-up input.email {\n display: block;\n box-sizing: border-box;\n width: 100%;\n}\n.sign-up input.button {\n margin-left: 8px;\n padding: 4px 8px;\n font: 600 13px \"Source Sans Pro\", sans-serif;\n text-transform: uppercase;\n letter-spacing: 1px;\n background: #ffbb33;\n border: none;\n transition: background-color 0.2s ease;\n}\n.sign-up input.button:hover {\n background: #ffd580;\n}\n.sign-up input:focus {\n border-color: #ffaa00;\n}\n\nbody, h1, h2, h3, h4, p, blockquote, code, ul, ol, dl, dd, img {\n margin: 0;\n}\n\nbody {\n background: #29313d url(\"image/background.png\") top center/100% auto no-repeat;\n color: #222;\n font: normal 16px/24px \"Crimson\", Georgia, serif;\n}\n\na {\n color: #1481b8;\n text-decoration: none;\n border-bottom: solid 1px rgba(222, 233, 237, 0);\n transition: color 0.2s ease, border-color 0.4s ease;\n}\n\na:hover {\n color: #1481b8;\n border-bottom: solid 1px #dee9ed;\n}\n\narticle {\n margin: 0 auto;\n padding: 0 0 12px 0;\n max-width: 960px;\n background: #fff;\n}\n\nheader {\n margin: 0 0 48px 0;\n color: #595959;\n background: #f5f3f0;\n border-bottom: solid 1px #dad8d6;\n}\n\nmain {\n margin: 0 48px;\n}\n\nimg.header {\n display: block;\n width: 100%;\n}\n\nimg.small {\n display: none;\n}\n\ndiv.intro {\n display: flex;\n}\ndiv.intro blockquote {\n flex-basis: 40%;\n margin: 0 48px 0 0;\n font: italic 28px/42px \"Crimson\", Georgia, serif;\n}\ndiv.intro div.text {\n flex-basis: 60%;\n margin: 8px 0 24px 0;\n}\n\np + p {\n margin-top: 24px;\n}\n\n.format {\n margin: 0 -12px 24px -12px;\n padding: 12px 12px 8px 12px;\n height: 244px;\n box-sizing: border-box;\n background: #eef4f7;\n background-size: cover;\n background-position: left;\n color: #444;\n border-radius: 3px;\n font: normal 16px/24px \"Source Sans Pro\", sans-serif;\n}\n.format h3 {\n margin: 0;\n padding: 0 0 4px 0;\n font: 600 16px/24px \"Source Sans Pro\", sans-serif;\n text-transform: uppercase;\n letter-spacing: 1px;\n}\n.format p {\n margin-bottom: 8px;\n}\n\n.format.print, .format.pdf {\n background-position: right;\n text-align: right;\n}\n\n.format-info {\n display: inline-block;\n width: 384px;\n text-align: left;\n}\n.format-info table {\n width: 100%;\n border-collapse: collapse;\n}\n.format-info table td + td {\n padding-left: 5px;\n}\n\n.format.print {\n background-image: url(\"image/format-print.jpg\");\n}\n\n.format.ebook {\n background-image: url(\"image/format-ebook.jpg\");\n}\n\n.format.pdf {\n background-image: url(\"image/format-pdf.jpg\");\n}\n\n.format.web {\n background-image: url(\"image/format-web.jpg\");\n}\n\na.action {\n display: block;\n margin: 0 0 4px 0;\n padding: 4px 0;\n text-align: center;\n border-radius: 3px;\n background: #1481b8;\n transition: background-color 0.2s ease, color 0.2s ease;\n font: 400 17px/24px \"Source Sans Pro\", sans-serif;\n color: white;\n}\na.action small {\n font-size: 14px;\n padding: 4px;\n color: rgba(255, 255, 255, 0.7);\n transition: color 0.2s ease;\n}\n\na.action:hover {\n background-color: #2badee;\n}\na.action:hover small {\n color: white;\n}\n\nh3 {\n font: italic 24px/24px \"Crimson\", Georgia, serif;\n margin: 12px 0;\n}\n\nimg.author {\n float: left;\n width: 240px;\n margin: 0 12px 0 -12px;\n padding: 12px;\n background: #f5f3f0;\n border-radius: 3px;\n}\n\ndiv.author {\n vertical-align: top;\n margin: 36px 0 0 288px;\n}\n\nfooter {\n position: relative;\n border-top: solid 1px #dee9ed;\n color: #7aa0b8;\n font: 400 15px \"Source Sans Pro\", sans-serif;\n text-align: center;\n margin: 12px 0 36px 0;\n padding-top: 48px;\n}\nfooter a, footer a:hover {\n border: none;\n}\n\n@media only screen and (max-width: 700px) {\n main {\n margin: 0 24px;\n }\n\n header {\n margin-bottom: 24px;\n }\n\n img.big {\n display: none;\n }\n\n img.small {\n display: block;\n }\n\n div.intro {\n display: block;\n }\n div.intro blockquote {\n display: block;\n font: italic 24px/36px \"Crimson\", Georgia, serif;\n }\n div.intro div.text {\n display: block;\n margin: 24px 0 24px 0;\n }\n\n .format {\n margin-bottom: 12px;\n height: auto;\n background-blend-mode: lighten;\n }\n\n .format-info {\n display: block;\n width: 100%;\n }\n\n .format.print {\n background-color: #a6a29f;\n }\n\n .format.ebook {\n background-color: #97a2aa;\n }\n\n .format.pdf {\n background-color: #cfccca;\n }\n\n .format.web {\n background-color: #d6dbd3;\n }\n\n img.author {\n float: none;\n }\n\n div.author {\n margin: 0 0 0 0;\n }\n}" }, { "path": "site/index.html", "content": "\n\n\n\nCrafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n\n
\n \"Crafting\"Crafting\n
\n\n
\n\n
\n\n

Ever wanted to make your own programming language or wondered how\nthey are designed and built?

If so, this book is for you.

\n\n
\n\n

Crafting Interpreters contains everything you need to implement a\nfull-featured, efficient scripting language. You’ll learn both high-level\nconcepts around parsing and semantics and gritty details like bytecode\nrepresentation and garbage collection. Your brain will light up with new ideas,\nand your hands will get dirty and calloused. It’s a blast.

\n\n

Starting from main(), you build a language that features rich\nsyntax, dynamic typing, garbage collection, lexical scope, first-class\nfunctions, closures, classes, and inheritance. All packed into a few thousand\nlines of clean, fast code that you thoroughly understand because you write each\none yourself.

\n\n

The book is available in four delectable formats:

\n\n
\n\n
\n\n
\n
\n

Print

\n

640 pages of beautiful typography and high resolution hand-drawn\n illustrations. Each page lovingly typeset by the author. The premiere reading\n experience.

\n \n \n \n \n \n \n \n \n \n \n \n \n
\n Amazon.com\n \n .ca\n \n .uk\n \n .au\n \n .de\n \n .fr\n \n .es\n \n .it\n \n .jp\n
\n \n \n \n \n \n
\n Barnes and Noble\n \n Book Depository\n
\n Download Sample PDF\n
\n
\n
\n
\n

eBook

\n

Carefully tuned CSS fits itself to your ebook reader and screen size.\n Full-color syntax highlighting and live hyperlinks. Like Alan Kay's Dynabook\n but real.

\n \n \n \n \n \n \n \n \n \n
\n Kindle Amazon.com\n \n .uk\n \n .ca\n \n .au\n \n .de\n \n .in\n
\n \n \n \n \n \n \n \n \n \n \n
\n .fr\n \n .es\n \n .it\n \n .jp\n \n .br\n \n .mx\n \n Apple Books\n
\n \n \n \n \n \n \n
\n Play Books Google\n \n Nook B&N\n \n EPUB Smashwords\n
\n
\n
\n
\n
\n

PDF

\n

Perfectly mirrors the hand-crafted typesetting and sharp illustrations of\n the print book, but much easier to carry around.

\n Buy from Payhip\n Download Free Sample\n
\n
\n
\n
\n

Web

\n

Meticulous responsive design looks great from your desktop down to your\n phone. Every chapter, aside, and illustration is there. Read the whole book\n for free. Really.

\n Read Now\n
\n
\n\n\n\n
\n

About Robert Nystrom

\n\n

I got bitten by the language bug years ago while on paternity leave between\nmidnight feedings. I cobbled together a number of hobby languages before worming my way into an honest-to-God,\nfull-time programming language job. Today, I work at Google on the Dart language.

\n\n

Before I fell in love with languages, I developed games at Electronic Arts\nfor eight years. I wrote the best-selling book Game Programming\nPatterns based on what I learned there. You can read that book for free\ntoo.

\n\n

If you want more, you can find me on Twitter (@munificentbob), email me at bob\nat this site's domain (though I am slow to respond), read my blog, or join\nmy low frequency mailing list:

\n\n
\n \n
\n
\n \n \n
\n \n
\n
\n \n
\n\n
\n\n\n
\n
\n\n\n" }, { "path": "site/inheritance.html", "content": "\n\n\n\nInheritance · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
13
\n

Inheritance

\n\n
\n

Once we were blobs in the sea, and then fishes, and then lizards and rats and\nthen monkeys, and hundreds of things in between. This hand was once a fin,\nthis hand once had claws! In my human mouth I have the pointy teeth of a wolf\nand the chisel teeth of a rabbit and the grinding teeth of a cow! Our blood is\nas salty as the sea we used to live in! When we’re frightened, the hair on our\nskin stands up, just like it did when we had fur. We are history! Everything\nwe’ve ever been on the way to becoming us, we still are.

\n

Terry Pratchett, A Hat Full of Sky

\n
\n

Can you believe it? We’ve reached the last chapter of Part II. We’re almost\ndone with our first Lox interpreter. The previous chapter was a big ball of\nintertwined object-orientation features. I couldn’t separate those from each\nother, but I did manage to untangle one piece. In this chapter, we’ll finish\noff Lox’s class support by adding inheritance.

\n

Inheritance appears in object-oriented languages all the way back to the first one, Simula. Early on, Kristen Nygaard and\nOle-Johan Dahl noticed commonalities across classes in the simulation programs\nthey wrote. Inheritance gave them a way to reuse the code for those similar\nparts.

\n\n

13 . 1Superclasses and Subclasses

\n

Given that the concept is “inheritance”, you would hope they would pick a\nconsistent metaphor and call them “parent” and “child” classes, but that would\nbe too easy. Way back when, C. A. R. Hoare coined the term “subclass” to refer to a record type that refines another\ntype. Simula borrowed that term to refer to a class that inherits from\nanother. I don’t think it was until Smalltalk came along that someone flipped\nthe Latin prefix to get “superclass” to refer to the other side of the\nrelationship. From C++, you also hear “base” and “derived” classes. I’ll mostly\nstick with “superclass” and “subclass”.

\n\n

Our first step towards supporting inheritance in Lox is a way to specify a\nsuperclass when declaring a class. There’s a lot of variety in syntax for this.\nC++ and C# place a : after the subclass’s name, followed by the superclass\nname. Java uses extends instead of the colon. Python puts the superclass(es)\nin parentheses after the class name. Simula puts the superclass’s name before\nthe class keyword.

\n

This late in the game, I’d rather not add a new reserved word or token to the\nlexer. We don’t have extends or even :, so we’ll follow Ruby and use a\nless-than sign (<).

\n
class Doughnut {\n  // General doughnut stuff...\n}\n\nclass BostonCream < Doughnut {\n  // Boston Cream-specific stuff...\n}\n
\n

To work this into the grammar, we add a new optional clause in our existing\nclassDecl rule.

\n
classDecl      → "class" IDENTIFIER ( "<" IDENTIFIER )?\n                 "{" function* "}" ;\n
\n

After the class name, you can have a < followed by the superclass’s name. The\nsuperclass clause is optional because you don’t have to have a superclass.\nUnlike some other object-oriented languages like Java, Lox has no root “Object”\nclass that everything inherits from, so when you omit the superclass clause, the\nclass has no superclass, not even an implicit one.

\n

We want to capture this new syntax in the class declaration’s AST node.

\n
      "Block      : List<Stmt> statements",\n
tool/GenerateAst.java
\nin main()
\nreplace 1 line
\n
      "Class      : Token name, Expr.Variable superclass," +\n                  " List<Stmt.Function> methods",\n
      "Expression : Expr expression",\n
\n
tool/GenerateAst.java, in main(), replace 1 line
\n\n

You might be surprised that we store the superclass name as an Expr.Variable,\nnot a Token. The grammar restricts the superclass clause to a single identifier,\nbut at runtime, that identifier is evaluated as a variable access. Wrapping the\nname in an Expr.Variable early on in the parser gives us an object that the\nresolver can hang the resolution information off of.

\n

The new parser code follows the grammar directly.

\n
    Token name = consume(IDENTIFIER, "Expect class name.");\n
lox/Parser.java
\nin classDeclaration()
\n
\n\n    Expr.Variable superclass = null;\n    if (match(LESS)) {\n      consume(IDENTIFIER, "Expect superclass name.");\n      superclass = new Expr.Variable(previous());\n    }\n\n
    consume(LEFT_BRACE, "Expect '{' before class body.");\n
\n
lox/Parser.java, in classDeclaration()
\n\n

Once we’ve (possibly) parsed a superclass declaration, we store it in the AST.

\n
    consume(RIGHT_BRACE, "Expect '}' after class body.");\n\n
lox/Parser.java
\nin classDeclaration()
\nreplace 1 line
\n
    return new Stmt.Class(name, superclass, methods);\n
  }\n
\n
lox/Parser.java, in classDeclaration(), replace 1 line
\n\n

If we didn’t parse a superclass clause, the superclass expression will be\nnull. We’ll have to make sure the later passes check for that. The first of\nthose is the resolver.

\n
    define(stmt.name);\n
lox/Resolver.java
\nin visitClassStmt()
\n
\n\n    if (stmt.superclass != null) {\n      resolve(stmt.superclass);\n    }\n
\n\n    beginScope();\n
\n
lox/Resolver.java, in visitClassStmt()
\n\n

The class declaration AST node has a new subexpression, so we traverse into and\nresolve that. Since classes are usually declared at the top level, the\nsuperclass name will most likely be a global variable, so this doesn’t usually\ndo anything useful. However, Lox allows class declarations even inside blocks,\nso it’s possible the superclass name refers to a local variable. In that case,\nwe need to make sure it’s resolved.

\n

Because even well-intentioned programmers sometimes write weird code, there’s a\nsilly edge case we need to worry about while we’re in here. Take a look at this:

\n
class Oops < Oops {}\n
\n

There’s no way this will do anything useful, and if we let the runtime try to\nrun this, it will break the expectation the interpreter has about there not\nbeing cycles in the inheritance chain. The safest thing is to detect this case\nstatically and report it as an error.

\n
    define(stmt.name);\n\n
lox/Resolver.java
\nin visitClassStmt()
\n
    if (stmt.superclass != null &&\n        stmt.name.lexeme.equals(stmt.superclass.name.lexeme)) {\n      Lox.error(stmt.superclass.name,\n          "A class can't inherit from itself.");\n    }\n\n
    if (stmt.superclass != null) {\n
\n
lox/Resolver.java, in visitClassStmt()
\n\n

Assuming the code resolves without error, the AST travels to the interpreter.

\n
  public Void visitClassStmt(Stmt.Class stmt) {\n
lox/Interpreter.java
\nin visitClassStmt()
\n
    Object superclass = null;\n    if (stmt.superclass != null) {\n      superclass = evaluate(stmt.superclass);\n      if (!(superclass instanceof LoxClass)) {\n        throw new RuntimeError(stmt.superclass.name,\n            "Superclass must be a class.");\n      }\n    }\n\n
    environment.define(stmt.name.lexeme, null);\n
\n
lox/Interpreter.java, in visitClassStmt()
\n\n

If the class has a superclass expression, we evaluate it. Since that could\npotentially evaluate to some other kind of object, we have to check at runtime\nthat the thing we want to be the superclass is actually a class. Bad things\nwould happen if we allowed code like:

\n
var NotAClass = "I am totally not a class";\n\nclass Subclass < NotAClass {} // ?!\n
\n

Assuming that check passes, we continue on. Executing a class declaration turns\nthe syntactic representation of a class—its AST node—into its runtime\nrepresentation, a LoxClass object. We need to plumb the superclass through to\nthat too. We pass the superclass to the constructor.

\n
      methods.put(method.name.lexeme, function);\n    }\n\n
lox/Interpreter.java
\nin visitClassStmt()
\nreplace 1 line
\n
    LoxClass klass = new LoxClass(stmt.name.lexeme,\n        (LoxClass)superclass, methods);\n\n
    environment.assign(stmt.name, klass);\n
\n
lox/Interpreter.java, in visitClassStmt(), replace 1 line
\n\n

The constructor stores it in a field.

\n
lox/LoxClass.java
\nconstructor LoxClass()
\nreplace 1 line
\n
  LoxClass(String name, LoxClass superclass,\n           Map<String, LoxFunction> methods) {\n    this.superclass = superclass;\n
    this.name = name;\n
\n
lox/LoxClass.java, constructor LoxClass(), replace 1 line
\n\n

Which we declare here:

\n
  final String name;\n
lox/LoxClass.java
\nin class LoxClass
\n
  final LoxClass superclass;\n
  private final Map<String, LoxFunction> methods;\n
\n
lox/LoxClass.java, in class LoxClass
\n\n

With that, we can define classes that are subclasses of other classes. Now, what\ndoes having a superclass actually do?

\n

13 . 2Inheriting Methods

\n

Inheriting from another class means that everything that’s true of the superclass should be true, more or less, of the\nsubclass. In statically typed languages, that carries a lot of implications. The\nsubclass must also be a subtype, and the memory layout is controlled so that\nyou can pass an instance of a subclass to a function expecting a superclass and\nit can still access the inherited fields correctly.

\n\n

Lox is a dynamically typed language, so our requirements are much simpler.\nBasically, it means that if you can call some method on an instance of the\nsuperclass, you should be able to call that method when given an instance of the\nsubclass. In other words, methods are inherited from the superclass.

\n

This lines up with one of the goals of inheritance—to give users a way to\nreuse code across classes. Implementing this in our interpreter is\nastonishingly easy.

\n
      return methods.get(name);\n    }\n\n
lox/LoxClass.java
\nin findMethod()
\n
    if (superclass != null) {\n      return superclass.findMethod(name);\n    }\n\n
    return null;\n
\n
lox/LoxClass.java, in findMethod()
\n\n

That’s literally all there is to it. When we are looking up a method on an\ninstance, if we don’t find it on the instance’s class, we recurse up through the\nsuperclass chain and look there. Give it a try:

\n
class Doughnut {\n  cook() {\n    print "Fry until golden brown.";\n  }\n}\n\nclass BostonCream < Doughnut {}\n\nBostonCream().cook();\n
\n

There we go, half of our inheritance features are complete with only three lines\nof Java code.

\n

13 . 3Calling Superclass Methods

\n

In findMethod() we look for a method on the current class before walking up\nthe superclass chain. If a method with the same name exists in both the subclass\nand the superclass, the subclass one takes precedence or overrides the\nsuperclass method. Sort of like how variables in inner scopes shadow outer ones.

\n

That’s great if the subclass wants to replace some superclass behavior\ncompletely. But, in practice, subclasses often want to refine the superclass’s\nbehavior. They want to do a little work specific to the subclass, but also\nexecute the original superclass behavior too.

\n

However, since the subclass has overridden the method, there’s no way to refer\nto the original one. If the subclass method tries to call it by name, it will\njust recursively hit its own override. We need a way to say “Call this method,\nbut look for it directly on my superclass and ignore my override”. Java uses\nsuper for this, and we’ll use that same syntax in Lox. Here is an example:

\n
class Doughnut {\n  cook() {\n    print "Fry until golden brown.";\n  }\n}\n\nclass BostonCream < Doughnut {\n  cook() {\n    super.cook();\n    print "Pipe full of custard and coat with chocolate.";\n  }\n}\n\nBostonCream().cook();\n
\n

If you run this, it should print:

\n
Fry until golden brown.\nPipe full of custard and coat with chocolate.\n
\n

We have a new expression form. The super keyword, followed by a dot and an\nidentifier, looks for a method with that name. Unlike calls on this, the search\nstarts at the superclass.

\n

13 . 3 . 1Syntax

\n

With this, the keyword works sort of like a magic variable, and the expression\nis that one lone token. But with super, the subsequent . and property name\nare inseparable parts of the super expression. You can’t have a bare super\ntoken all by itself.

\n
print super; // Syntax error.\n
\n

So the new clause we add to the primary rule in our grammar includes the\nproperty access as well.

\n
primary        → "true" | "false" | "nil" | "this"\n               | NUMBER | STRING | IDENTIFIER | "(" expression ")"\n               | "super" "." IDENTIFIER ;\n
\n

Typically, a super expression is used for a method call, but, as with regular\nmethods, the argument list is not part of the expression. Instead, a super\ncall is a super access followed by a function call. Like other method calls,\nyou can get a handle to a superclass method and invoke it separately.

\n
var method = super.cook;\nmethod();\n
\n

So the super expression itself contains only the token for the super keyword\nand the name of the method being looked up. The corresponding syntax tree node is thus:

\n
      "Set      : Expr object, Token name, Expr value",\n
tool/GenerateAst.java
\nin main()
\n
      "Super    : Token keyword, Token method",\n
      "This     : Token keyword",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

Following the grammar, the new parsing code goes inside our existing primary()\nmethod.

\n
      return new Expr.Literal(previous().literal);\n    }\n
lox/Parser.java
\nin primary()
\n
\n\n    if (match(SUPER)) {\n      Token keyword = previous();\n      consume(DOT, "Expect '.' after 'super'.");\n      Token method = consume(IDENTIFIER,\n          "Expect superclass method name.");\n      return new Expr.Super(keyword, method);\n    }\n
\n\n    if (match(THIS)) return new Expr.This(previous());\n
\n
lox/Parser.java, in primary()
\n\n

A leading super keyword tells us we’ve hit a super expression. After that we\nconsume the expected . and method name.

\n

13 . 3 . 2Semantics

\n

Earlier, I said a super expression starts the method lookup from “the\nsuperclass”, but which superclass? The naïve answer is the superclass of\nthis, the object the surrounding method was called on. That coincidentally\nproduces the right behavior in a lot of cases, but that’s not actually correct.\nGaze upon:

\n
class A {\n  method() {\n    print "A method";\n  }\n}\n\nclass B < A {\n  method() {\n    print "B method";\n  }\n\n  test() {\n    super.method();\n  }\n}\n\nclass C < B {}\n\nC().test();\n
\n

Translate this program to Java, C#, or C++ and it will print “A method”, which\nis what we want Lox to do too. When this program runs, inside the body of\ntest(), this is an instance of C. The superclass of C is B, but that is\nnot where the lookup should start. If it did, we would hit B’s method().

\n

Instead, lookup should start on the superclass of the class containing the\nsuper expression. In this case, since test() is defined inside B, the\nsuper expression inside it should start the lookup on B’s superclass—A.

\n

\"The\n\n

Thus, in order to evaluate a super expression, we need access to the\nsuperclass of the class definition surrounding the call. Alack and alas, at the\npoint in the interpreter where we are executing a super expression, we don’t\nhave that easily available.

\n

We could add a field to LoxFunction to store a reference to the LoxClass that\nowns that method. The interpreter would keep a reference to the\ncurrently executing LoxFunction so that we could look it up later when we hit a\nsuper expression. From there, we’d get the LoxClass of the method, then its\nsuperclass.

\n

That’s a lot of plumbing. In the last chapter, we had a similar problem when\nwe needed to add support for this. In that case, we used our existing\nenvironment and closure mechanism to store a reference to the current object.\nCould we do something similar for storing the superclass? Well, I probably wouldn’t be talking about it if the\nanswer was no, so . . . yes.

\n\n

One important difference is that we bound this when the method was accessed.\nThe same method can be called on different instances and each needs its own\nthis. With super expressions, the superclass is a fixed property of the\nclass declaration itself. Every time you evaluate some super expression, the\nsuperclass is always the same.

\n

That means we can create the environment for the superclass once, when the class\ndefinition is executed. Immediately before we define the methods, we make a new\nenvironment to bind the class’s superclass to the name super.

\"The\n

When we create the LoxFunction runtime representation for each method, that is\nthe environment they will capture in their closure. Later, when a method is\ninvoked and this is bound, the superclass environment becomes the parent for\nthe method’s environment, like so:

\"The\n

That’s a lot of machinery, but we’ll get through it a step at a time. Before we\ncan get to creating the environment at runtime, we need to handle the\ncorresponding scope chain in the resolver.

\n
      resolve(stmt.superclass);\n    }\n
lox/Resolver.java
\nin visitClassStmt()
\n
\n\n    if (stmt.superclass != null) {\n      beginScope();\n      scopes.peek().put("super", true);\n    }\n
\n\n    beginScope();\n
\n
lox/Resolver.java, in visitClassStmt()
\n\n

If the class declaration has a superclass, then we create a new scope\nsurrounding all of its methods. In that scope, we define the name “super”. Once\nwe’re done resolving the class’s methods, we discard that scope.

\n
    endScope();\n\n
lox/Resolver.java
\nin visitClassStmt()
\n
    if (stmt.superclass != null) endScope();\n\n
    currentClass = enclosingClass;\n
\n
lox/Resolver.java, in visitClassStmt()
\n\n

It’s a minor optimization, but we only create the superclass environment if the\nclass actually has a superclass. There’s no point creating it when there isn’t\na superclass since there’d be no superclass to store in it anyway.

\n

With “super” defined in a scope chain, we are able to resolve the super\nexpression itself.

\n
lox/Resolver.java
\nadd after visitSetExpr()
\n
  @Override\n  public Void visitSuperExpr(Expr.Super expr) {\n    resolveLocal(expr, expr.keyword);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitSetExpr()
\n\n

We resolve the super token exactly as if it were a variable. The resolution\nstores the number of hops along the environment chain that the interpreter needs\nto walk to find the environment where the superclass is stored.

\n

This code is mirrored in the interpreter. When we evaluate a subclass\ndefinition, we create a new environment.

\n
        throw new RuntimeError(stmt.superclass.name,\n            "Superclass must be a class.");\n      }\n    }\n\n    environment.define(stmt.name.lexeme, null);\n
lox/Interpreter.java
\nin visitClassStmt()
\n
\n\n    if (stmt.superclass != null) {\n      environment = new Environment(environment);\n      environment.define("super", superclass);\n    }\n
\n\n    Map<String, LoxFunction> methods = new HashMap<>();\n
\n
lox/Interpreter.java, in visitClassStmt()
\n\n

Inside that environment, we store a reference to the superclass—the actual\nLoxClass object for the superclass which we have now that we are in the runtime.\nThen we create the LoxFunctions for each method. Those will capture the current\nenvironment—the one where we just bound “super”—as their closure, holding\non to the superclass like we need. Once that’s done, we pop the environment.

\n
    LoxClass klass = new LoxClass(stmt.name.lexeme,\n        (LoxClass)superclass, methods);\n
lox/Interpreter.java
\nin visitClassStmt()
\n
\n\n    if (superclass != null) {\n      environment = environment.enclosing;\n    }\n
\n\n    environment.assign(stmt.name, klass);\n
\n
lox/Interpreter.java, in visitClassStmt()
\n\n

We’re ready to interpret super expressions themselves. There are a few moving\nparts, so we’ll build this method up in pieces.

\n
lox/Interpreter.java
\nadd after visitSetExpr()
\n
  @Override\n  public Object visitSuperExpr(Expr.Super expr) {\n    int distance = locals.get(expr);\n    LoxClass superclass = (LoxClass)environment.getAt(\n        distance, "super");\n  }\n
\n
lox/Interpreter.java, add after visitSetExpr()
\n\n

First, the work we’ve been leading up to. We look up the surrounding class’s\nsuperclass by looking up “super” in the proper environment.

\n

When we access a method, we also need to bind this to the object the method is\naccessed from. In an expression like doughnut.cook, the object is whatever we\nget from evaluating doughnut. In a super expression like super.cook, the\ncurrent object is implicitly the same current object that we’re using. In\nother words, this. Even though we are looking up the method on the\nsuperclass, the instance is still this.

\n

Unfortunately, inside the super expression, we don’t have a convenient node\nfor the resolver to hang the number of hops to this on. Fortunately, we do\ncontrol the layout of the environment chains. The environment where “this” is\nbound is always right inside the environment where we store “super”.

\n
    LoxClass superclass = (LoxClass)environment.getAt(\n        distance, "super");\n
lox/Interpreter.java
\nin visitSuperExpr()
\n
\n\n    LoxInstance object = (LoxInstance)environment.getAt(\n        distance - 1, "this");\n
  }\n
\n
lox/Interpreter.java, in visitSuperExpr()
\n\n

Offsetting the distance by one looks up “this” in that inner environment. I\nadmit this isn’t the most elegant code, but it\nworks.

\n\n

Now we’re ready to look up and bind the method, starting at the superclass.

\n
    LoxInstance object = (LoxInstance)environment.getAt(\n        distance - 1, "this");\n
lox/Interpreter.java
\nin visitSuperExpr()
\n
\n\n    LoxFunction method = superclass.findMethod(expr.method.lexeme);\n    return method.bind(object);\n
  }\n
\n
lox/Interpreter.java, in visitSuperExpr()
\n\n

This is almost exactly like the code for looking up a method of a get\nexpression, except that we call findMethod() on the superclass instead of on\nthe class of the current object.

\n

That’s basically it. Except, of course, that we might fail to find the method.\nSo we check for that too.

\n
\n\n    LoxFunction method = superclass.findMethod(expr.method.lexeme);\n
lox/Interpreter.java
\nin visitSuperExpr()
\n
\n\n    if (method == null) {\n      throw new RuntimeError(expr.method,\n          "Undefined property '" + expr.method.lexeme + "'.");\n    }\n\n
    return method.bind(object);\n  }\n
\n
lox/Interpreter.java, in visitSuperExpr()
\n\n

There you have it! Take that BostonCream example earlier and give it a try.\nAssuming you and I did everything right, it should fry it first, then stuff it\nwith cream.

\n

13 . 3 . 3Invalid uses of super

\n

As with previous language features, our implementation does the right thing when\nthe user writes correct code, but we haven’t bulletproofed the intepreter\nagainst bad code. In particular, consider:

\n
class Eclair {\n  cook() {\n    super.cook();\n    print "Pipe full of crème pâtissière.";\n  }\n}\n
\n

This class has a super expression, but no superclass. At runtime, the code for\nevaluating super expressions assumes that “super” was successfully resolved\nand will be found in the environment. That’s going to fail here because there is\nno surrounding environment for the superclass since there is no superclass. The\nJVM will throw an exception and bring our interpreter to its knees.

\n

Heck, there are even simpler broken uses of super:

\n
super.notEvenInAClass();\n
\n

We could handle errors like these at runtime by checking to see if the lookup\nof “super” succeeded. But we can tell statically—just by looking at the\nsource code—that Eclair has no superclass and thus no super expression will\nwork inside it. Likewise, in the second example, we know that the super\nexpression is not even inside a method body.

\n

Even though Lox is dynamically typed, that doesn’t mean we want to defer\neverything to runtime. If the user made a mistake, we’d like to help them find\nit sooner rather than later. So we’ll report these errors statically, in the\nresolver.

\n

First, we add a new case to the enum we use to keep track of what kind of class\nis surrounding the current code being visited.

\n
    NONE,\n
    CLASS,\n
lox/Resolver.java
\nin enum ClassType
\nadd “,” to previous line
\n
    SUBCLASS\n
  }\n
\n
lox/Resolver.java, in enum ClassType, add “,” to previous line
\n\n

We’ll use that to distinguish when we’re inside a class that has a superclass\nversus one that doesn’t. When we resolve a class declaration, we set that if the\nclass is a subclass.

\n
    if (stmt.superclass != null) {\n
lox/Resolver.java
\nin visitClassStmt()
\n
      currentClass = ClassType.SUBCLASS;\n
      resolve(stmt.superclass);\n
\n
lox/Resolver.java, in visitClassStmt()
\n\n

Then, when we resolve a super expression, we check to see that we are\ncurrently inside a scope where that’s allowed.

\n
  public Void visitSuperExpr(Expr.Super expr) {\n
lox/Resolver.java
\nin visitSuperExpr()
\n
    if (currentClass == ClassType.NONE) {\n      Lox.error(expr.keyword,\n          "Can't use 'super' outside of a class.");\n    } else if (currentClass != ClassType.SUBCLASS) {\n      Lox.error(expr.keyword,\n          "Can't use 'super' in a class with no superclass.");\n    }\n\n
    resolveLocal(expr, expr.keyword);\n
\n
lox/Resolver.java, in visitSuperExpr()
\n\n

If not—oopsie!—the user made a mistake.

\n

13 . 4Conclusion

\n

We made it! That final bit of error handling is the last chunk of code needed to\ncomplete our Java implementation of Lox. This is a real accomplishment and one you should be proud of. In the\npast dozen chapters and a thousand or so lines of code, we have learned and\nimplemented . . . 

\n\n\n

We did all of that from scratch, with no external dependencies or magic tools.\nJust you and I, our respective text editors, a couple of collection classes in\nthe Java standard library, and the JVM runtime.

\n

This marks the end of Part II, but not the end of the book. Take a break. Maybe\nwrite a few fun Lox programs and run them in your interpreter. (You may want to\nadd a few more native methods for things like reading user input.) When you’re\nrefreshed and ready, we’ll embark on our next adventure.

\n
\n

Challenges

\n
    \n
  1. \n

    Lox supports only single inheritance—a class may have a single\nsuperclass and that’s the only way to reuse methods across classes. Other\nlanguages have explored a variety of ways to more freely reuse and share\ncapabilities across classes: mixins, traits, multiple inheritance, virtual\ninheritance, extension methods, etc.

    \n

    If you were to add some feature along these lines to Lox, which would you\npick and why? If you’re feeling courageous (and you should be at this\npoint), go ahead and add it.

    \n
    2. \n
  2. \n

    In Lox, as in most other object-oriented languages, when looking up a\nmethod, we start at the bottom of the class hierarchy and work our way up—a subclass’s method is preferred over a superclass’s. In order to get to the\nsuperclass method from within an overriding method, you use super.

    \n

    The language BETA takes the opposite approach. When you call a\nmethod, it starts at the top of the class hierarchy and works down. A\nsuperclass method wins over a subclass method. In order to get to the\nsubclass method, the superclass method can call inner, which is sort of\nlike the inverse of super. It chains to the next method down the\nhierarchy.

    \n

    The superclass method controls when and where the subclass is allowed to\nrefine its behavior. If the superclass method doesn’t call inner at all,\nthen the subclass has no way of overriding or modifying the superclass’s\nbehavior.

    \n

    Take out Lox’s current overriding and super behavior and replace it with\nBETA’s semantics. In short:

    \n
      \n
    • \n

      When calling a method on a class, prefer the method highest on the\nclass’s inheritance chain.

      \n
      • \n
    • \n

      Inside the body of a method, a call to inner looks for a method with\nthe same name in the nearest subclass along the inheritance chain\nbetween the class containing the inner and the class of this. If\nthere is no matching method, the inner call does nothing.

      \n
      • \n
    \n

    For example:

    \n
    class Doughnut {\n  cook() {\n    print "Fry until golden brown.";\n    inner();\n    print "Place in a nice box.";\n  }\n}\n\nclass BostonCream < Doughnut {\n  cook() {\n    print "Pipe full of custard and coat with chocolate.";\n  }\n}\n\nBostonCream().cook();\n
    \n

    This should print:

    \n
    Fry until golden brown.\nPipe full of custard and coat with chocolate.\nPlace in a nice box.\n
    \n
    3. \n
  3. \n

    In the chapter where I introduced Lox, I challenged you to\ncome up with a couple of features you think the language is missing. Now\nthat you know how to build an interpreter, implement one of those features.

    \n
    4. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/introduction.html", "content": "\n\n\n\nIntroduction · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
1
\n

Introduction

\n\n
\n

Fairy tales are more than true: not because they tell us that dragons exist,\nbut because they tell us that dragons can be beaten.

\n

G.K. Chesterton by way of Neil Gaiman, Coraline

\n
\n

I’m really excited we’re going on this journey together. This is a book on\nimplementing interpreters for programming languages. It’s also a book on how to\ndesign a language worth implementing. It’s the book I wish I’d had when I first\nstarted getting into languages, and it’s the book I’ve been writing in my head for nearly a decade.

\n\n

In these pages, we will walk step-by-step through two complete interpreters for\na full-featured language. I assume this is your first foray into languages, so\nI’ll cover each concept and line of code you need to build a complete, usable,\nfast language implementation.

\n

In order to cram two full implementations inside one book without it turning\ninto a doorstop, this text is lighter on theory than others. As we build each\npiece of the system, I will introduce the history and concepts behind it. I’ll\ntry to get you familiar with the lingo so that if you ever find yourself at a\ncocktail party full of PL (programming language)\nresearchers, you’ll fit in.

\n\n

But we’re mostly going to spend our brain juice getting the language up and\nrunning. This is not to say theory isn’t important. Being able to reason\nprecisely and formally about syntax and semantics is\na vital skill when working on a language. But, personally, I learn best by\ndoing. It’s hard for me to wade through paragraphs full of abstract concepts and\nreally absorb them. But if I’ve coded something, run it, and debugged it, then I\nget it.

\n\n

That’s my goal for you. I want you to come away with a solid intuition of how a\nreal language lives and breathes. My hope is that when you read other, more\ntheoretical books later, the concepts there will firmly stick in your mind,\nadhered to this tangible substrate.

\n

1 . 1Why Learn This Stuff?

\n

Every introduction to every compiler book seems to have this section. I don’t\nknow what it is about programming languages that causes such existential doubt.\nI don’t think ornithology books worry about justifying their existence. They\nassume the reader loves birds and start teaching.

\n

But programming languages are a little different. I suppose it is true that the\nodds of any of us creating a broadly successful, general-purpose programming\nlanguage are slim. The designers of the world’s widely used languages could fit\nin a Volkswagen bus, even without putting the pop-top camper up. If joining that\nelite group was the only reason to learn languages, it would be hard to\njustify. Fortunately, it isn’t.

\n

1 . 1 . 1Little languages are everywhere

\n

For every successful general-purpose language, there are a thousand successful\nniche ones. We used to call them “little languages”, but inflation in the jargon\neconomy led to the name “domain-specific languages”. These are pidgins\ntailor-built to a specific task. Think application scripting languages, template\nengines, markup formats, and configuration files.

\n

\"A

\n\n

Almost every large software project needs a handful of these. When you can, it’s\ngood to reuse an existing one instead of rolling your own. Once you factor in\ndocumentation, debuggers, editor support, syntax highlighting, and all of the\nother trappings, doing it yourself becomes a tall order.

\n

But there’s still a good chance you’ll find yourself needing to whip up a parser\nor other tool when there isn’t an existing library that fits your needs. Even\nwhen you are reusing some existing implementation, you’ll inevitably end up\nneeding to debug and maintain it and poke around in its guts.

\n

1 . 1 . 2Languages are great exercise

\n

Long distance runners sometimes train with weights strapped to their ankles or\nat high altitudes where the atmosphere is thin. When they later unburden\nthemselves, the new relative ease of light limbs and oxygen-rich air enables\nthem to run farther and faster.

\n

Implementing a language is a real test of programming skill. The code is complex\nand performance critical. You must master recursion, dynamic arrays, trees,\ngraphs, and hash tables. You probably use hash tables at least in your\nday-to-day programming, but do you really understand them? Well, after we’ve\ncrafted our own from scratch, I guarantee you will.

\n

While I intend to show you that an interpreter isn’t as daunting as you might\nbelieve, implementing one well is still a challenge. Rise to it, and you’ll come\naway a stronger programmer, and smarter about how you use data structures and\nalgorithms in your day job.

\n

1 . 1 . 3One more reason

\n

This last reason is hard for me to admit, because it’s so close to my heart.\nEver since I learned to program as a kid, I felt there was something magical\nabout languages. When I first tapped out BASIC programs one key at a time I\ncouldn’t conceive how BASIC itself was made.

\n

Later, the mixture of awe and terror on my college friends’ faces when talking\nabout their compilers class was enough to convince me language hackers were a\ndifferent breed of human—some sort of wizards granted privileged access to\narcane arts.

\n

It’s a charming image, but it has a darker side. I\ndidn’t feel like a wizard, so I was left thinking I lacked some inborn quality\nnecessary to join the cabal. Though I’ve been fascinated by languages ever since\nI doodled made-up keywords in my school notebook, it took me decades to muster\nthe courage to try to really learn them. That “magical” quality, that sense of\nexclusivity, excluded me.

\n\n

When I did finally start cobbling together my own little interpreters, I quickly\nlearned that, of course, there is no magic at all. It’s just code, and the\npeople who hack on languages are just people.

\n

There are a few techniques you don’t often encounter outside of languages, and\nsome parts are a little difficult. But not more difficult than other obstacles\nyou’ve overcome. My hope is that if you’ve felt intimidated by languages and\nthis book helps you overcome that fear, maybe I’ll leave you just a tiny bit\nbraver than you were before.

\n

And, who knows, maybe you will make the next great language. Someone has to.

\n

1 . 2How the Book Is Organized

\n

This book is broken into three parts. You’re reading the first one now. It’s a\ncouple of chapters to get you oriented, teach you some of the lingo that\nlanguage hackers use, and introduce you to Lox, the language we’ll be\nimplementing.

\n

Each of the other two parts builds one complete Lox interpreter. Within those\nparts, each chapter is structured the same way. The chapter takes a single\nlanguage feature, teaches you the concepts behind it, and walks you through an\nimplementation.

\n

It took a good bit of trial and error on my part, but I managed to carve up the\ntwo interpreters into chapter-sized chunks that build on the previous chapters\nbut require nothing from later ones. From the very first chapter, you’ll have a\nworking program you can run and play with. With each passing chapter, it grows\nincreasingly full-featured until you eventually have a complete language.

\n

Aside from copious, scintillating English prose, chapters have a few other\ndelightful facets:

\n

1 . 2 . 1The code

\n

We’re about crafting interpreters, so this book contains real code. Every\nsingle line of code needed is included, and each snippet tells you where to\ninsert it in your ever-growing implementation.

\n

Many other language books and language implementations use tools like Lex\nand Yacc, so-called compiler-compilers, that\nautomatically generate some of the source files for an implementation from some\nhigher-level description. There are pros and cons to tools like those, and\nstrong opinions—some might say religious convictions—on both sides.

\n\n

We will abstain from using them here. I want to ensure there are no dark corners\nwhere magic and confusion can hide, so we’ll write everything by hand. As you’ll\nsee, it’s not as bad as it sounds, and it means you really will understand each\nline of code and how both interpreters work.

\n

A book has different constraints from the “real world” and so the coding style\nhere might not always reflect the best way to write maintainable production\nsoftware. If I seem a little cavalier about, say, omitting private or\ndeclaring a global variable, understand I do so to keep the code easier on your\neyes. The pages here aren’t as wide as your IDE and every character counts.

\n

Also, the code doesn’t have many comments. That’s because each handful of lines\nis surrounded by several paragraphs of honest-to-God prose explaining it. When\nyou write a book to accompany your program, you are welcome to omit comments\ntoo. Otherwise, you should probably use // a little more than I do.

\n

While the book contains every line of code and teaches what each means, it does\nnot describe the machinery needed to compile and run the interpreter. I assume\nyou can slap together a makefile or a project in your IDE of choice in order to\nget the code to run. Those kinds of instructions get out of date quickly, and\nI want this book to age like XO brandy, not backyard hooch.

\n

1 . 2 . 2Snippets

\n

Since the book contains literally every line of code needed for the\nimplementations, the snippets are quite precise. Also, because I try to keep the\nprogram in a runnable state even when major features are missing, sometimes we\nadd temporary code that gets replaced in later snippets.

\n

A snippet with all the bells and whistles looks like this:

\n
\n      default:\n
lox/Scanner.java
\nin scanToken()
\nreplace 1 line
\n
\n        if (isDigit(c)) {\n          number();\n        } else {\n          Lox.error(line, "Unexpected character.");\n        }\n
\n        break;\n
\n
lox/Scanner.java, in scanToken(), replace 1 line
\n

In the center, you have the new code to add. It may have a few faded out lines\nabove or below to show where it goes in the existing surrounding code. There is\nalso a little blurb telling you in which file and where to place the snippet. If\nthat blurb says “replace _ lines”, there is some existing code between the faded\nlines that you need to remove and replace with the new snippet.

\n

1 . 2 . 3Asides

\n

Asides contain biographical sketches, historical\nbackground, references to related topics, and suggestions of other areas to\nexplore. There’s nothing that you need to know in them to understand later\nparts of the book, so you can skip them if you want. I won’t judge you, but I\nmight be a little sad.

\n\n

1 . 2 . 4Challenges

\n

Each chapter ends with a few exercises. Unlike textbook problem sets, which tend\nto review material you already covered, these are to help you learn more than\nwhat’s in the chapter. They force you to step off the guided path and explore on\nyour own. They will make you research other languages, figure out how to\nimplement features, or otherwise get you out of your comfort zone.

\n

Vanquish the challenges and you’ll come away with a\nbroader understanding and possibly a few bumps and scrapes. Or skip them if you\nwant to stay inside the comfy confines of the tour bus. It’s your book.

\n\n

1 . 2 . 5Design notes

\n

Most “programming language” books are strictly programming language\nimplementation books. They rarely discuss how one might happen to design the\nlanguage being implemented. Implementation is fun because it is so precisely defined. We programmers seem to have an\naffinity for things that are black and white, ones and zeroes.

\n\n

Personally, I think the world needs only so many implementations of FORTRAN 77. At some point, you find yourself designing a\nnew language. Once you start playing that game, then the softer, human side\nof the equation becomes paramount. Things like which features are easy to learn,\nhow to balance innovation and familiarity, what syntax is more readable and to\nwhom.

\n\n

All of that stuff profoundly affects the success of your new language. I want\nyour language to succeed, so in some chapters I end with a “design note”, a\nlittle essay on some corner of the human aspect of programming languages. I’m no\nexpert on this—I don’t know if anyone really is—so take these with a large\npinch of salt. That should make them tastier food for thought, which is my main\naim.

\n

1 . 3The First Interpreter

\n

We’ll write our first interpreter, jlox, in Java. The\nfocus is on concepts. We’ll write the simplest, cleanest code we can to\ncorrectly implement the semantics of the language. This will get us comfortable\nwith the basic techniques and also hone our understanding of exactly how the\nlanguage is supposed to behave.

\n\n

Java is a great language for this. It’s high level enough that we don’t get\noverwhelmed by fiddly implementation details, but it’s still pretty explicit.\nUnlike in scripting languages, there tends to be less complex machinery hiding\nunder the hood, and you’ve got static types to see what data structures you’re\nworking with.

\n

I also chose Java specifically because it is an object-oriented language. That\nparadigm swept the programming world in the ’90s and is now the dominant way of\nthinking for millions of programmers. Odds are good you’re already used to\norganizing code into classes and methods, so we’ll keep you in that comfort\nzone.

\n

While academic language folks sometimes look down on object-oriented languages,\nthe reality is that they are widely used even for language work. GCC and LLVM\nare written in C++, as are most JavaScript virtual machines. Object-oriented\nlanguages are ubiquitous, and the tools and compilers for a language are often\nwritten in the same language.

\n\n

And, finally, Java is hugely popular. That means there’s a good chance you\nalready know it, so there’s less for you to learn to get going in the book. If\nyou aren’t that familiar with Java, don’t freak out. I try to stick to a fairly\nminimal subset of it. I use the diamond operator from Java 7 to make things a\nlittle more terse, but that’s about it as far as “advanced” features go. If you\nknow another object-oriented language, like C# or C++, you can muddle through.

\n

By the end of part II, we’ll have a simple, readable implementation. It’s not\nvery fast, but it’s correct. However, we are only able to accomplish that by\nbuilding on the Java virtual machine’s own runtime facilities. We want to learn\nhow Java itself implements those things.

\n

1 . 4The Second Interpreter

\n

So in the next part, we start all over again, but this time in C. C is the\nperfect language for understanding how an implementation really works, all the\nway down to the bytes in memory and the code flowing through the CPU.

\n

A big reason that we’re using C is so I can show you things C is particularly\ngood at, but that does mean you’ll need to be pretty comfortable with it. You\ndon’t have to be the reincarnation of Dennis Ritchie, but you shouldn’t be\nspooked by pointers either.

\n

If you aren’t there yet, pick up an introductory book on C and chew through it,\nthen come back here when you’re done. In return, you’ll come away from this book\nan even stronger C programmer. That’s useful given how many language\nimplementations are written in C: Lua, CPython, and Ruby’s MRI, to name a few.

\n

In our C interpreter, clox, we are forced to implement\nfor ourselves all the things Java gave us for free. We’ll write our own dynamic\narray and hash table. We’ll decide how objects are represented in memory, and\nbuild a garbage collector to reclaim them.

\n\n

Our Java implementation was focused on being correct. Now that we have that\ndown, we’ll turn to also being fast. Our C interpreter will contain a compiler that translates Lox to an efficient bytecode\nrepresentation (don’t worry, I’ll get into what that means soon), which it then\nexecutes. This is the same technique used by implementations of Lua, Python,\nRuby, PHP, and many other successful languages.

\n\n

We’ll even try our hand at benchmarking and optimization. By the end, we’ll have\na robust, accurate, fast interpreter for our language, able to keep up with\nother professional caliber implementations out there. Not bad for one book and a\nfew thousand lines of code.

\n
\n

Challenges

\n
    \n
  1. \n

    There are at least six domain-specific languages used in the little system\nI cobbled together to write and publish this book. What are they?

    \n
    2. \n
  2. \n

    Get a “Hello, world!” program written and running in Java. Set up whatever\nmakefiles or IDE projects you need to get it working. If you have a\ndebugger, get comfortable with it and step through your program as it runs.

    \n
    3. \n
  3. \n

    Do the same thing for C. To get some practice with pointers, define a\ndoubly linked list of heap-allocated strings. Write functions to insert,\nfind, and delete items from it. Test them.

    \n
    4. \n
\n
\n
\n

Design Note: What’s in a Name?

\n

One of the hardest challenges in writing this book was coming up with a name for\nthe language it implements. I went through pages of candidates before I found\none that worked. As you’ll discover on the first day you start building your own\nlanguage, naming is deviously hard. A good name satisfies a few criteria:

\n
    \n
  1. \n

    It isn’t in use. You can run into all sorts of trouble, legal and\nsocial, if you inadvertently step on someone else’s name.

    \n
    2. \n
  2. \n

    It’s easy to pronounce. If things go well, hordes of people will be\nsaying and writing your language’s name. Anything longer than a couple of\nsyllables or a handful of letters will annoy them to no end.

    \n
    3. \n
  3. \n

    It’s distinct enough to search for. People will Google your language’s\nname to learn about it, so you want a word that’s rare enough that most\nresults point to your docs. Though, with the amount of AI search engines are\npacking today, that’s less of an issue. Still, you won’t be doing your users\nany favors if you name your language “for”.

    \n
    4. \n
  4. \n

    It doesn’t have negative connotations across a number of cultures. This\nis hard to be on guard for, but it’s worth considering. The designer of\nNimrod ended up renaming his language to “Nim” because too many people\nremember that Bugs Bunny used “Nimrod” as an insult. (Bugs was using it\nironically.)

    \n
    5. \n
\n

If your potential name makes it through that gauntlet, keep it. Don’t get hung\nup on trying to find an appellation that captures the quintessence of your\nlanguage. If the names of the world’s other successful languages teach us\nanything, it’s that the name doesn’t matter much. All you need is a reasonably\nunique token.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/jumping-back-and-forth.html", "content": "\n\n\n\nJumping Back and Forth · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
23
\n

Jumping Back and Forth

\n\n
\n

The order that our mind imagines is like a net, or like a ladder, built to\nattain something. But afterward you must throw the ladder away, because you\ndiscover that, even if it was useful, it was meaningless.

\n

Umberto Eco, The Name of the Rose

\n
\n

It’s taken a while to get here, but we’re finally ready to add control flow to\nour virtual machine. In the tree-walk interpreter we built for jlox, we\nimplemented Lox’s control flow in terms of Java’s. To execute a Lox if\nstatement, we used a Java if statement to run the chosen branch. That works,\nbut isn’t entirely satisfying. By what magic does the JVM itself or a native\nCPU implement if statements? Now that we have our own bytecode VM to hack on,\nwe can answer that.

\n

When we talk about “control flow”, what are we referring to? By “flow” we mean\nthe way execution moves through the text of the program. Almost like there is a\nlittle robot inside the computer wandering through our code, executing bits and\npieces here and there. Flow is the path that robot takes, and by controlling\nthe robot, we drive which pieces of code it executes.

\n

In jlox, the robot’s locus of attention—the current bit of code—was\nimplicit based on which AST nodes were stored in various Java variables and what\nJava code we were in the middle of running. In clox, it is much more explicit.\nThe VM’s ip field stores the address of the current bytecode instruction. The\nvalue of that field is exactly “where we are” in the program.

\n

Execution proceeds normally by incrementing the ip. But we can mutate that\nvariable however we want to. In order to implement control flow, all that’s\nnecessary is to change the ip in more interesting ways. The simplest control\nflow construct is an if statement with no else clause:

\n
if (condition) print("condition was truthy");\n
\n

The VM evaluates the bytecode for the condition expression. If the result is\ntruthy, then it continues along and executes the print statement in the body.\nThe interesting case is when the condition is falsey. When that happens,\nexecution skips over the then branch and proceeds to the next statement.

\n

To skip over a chunk of code, we simply set the ip field to the address of the\nbytecode instruction following that code. To conditionally skip over some\ncode, we need an instruction that looks at the value on top of the stack. If\nit’s falsey, it adds a given offset to the ip to jump over a range of\ninstructions. Otherwise, it does nothing and lets execution proceed to the next\ninstruction as usual.

\n

When we compile to bytecode, the explicit nested block structure of the code\nevaporates, leaving only a flat series of instructions behind. Lox is a\nstructured programming language, but clox bytecode isn’t. The right—or\nwrong, depending on how you look at it—set of bytecode instructions could\njump into the middle of a block, or from one scope into another.

\n

The VM will happily execute that, even if the result leaves the stack in an\nunknown, inconsistent state. So even though the bytecode is unstructured, we’ll\ntake care to ensure that our compiler only generates clean code that maintains\nthe same structure and nesting that Lox itself does.

\n

This is exactly how real CPUs behave. Even though we might program them using\nhigher-level languages that mandate structured control flow, the compiler lowers\nthat down to raw jumps. At the bottom, it turns out goto is the only real\ncontrol flow.

\n

Anyway, I didn’t mean to get all philosophical. The important bit is that if we\nhave that one conditional jump instruction, that’s enough to implement Lox’s\nif statement, as long as it doesn’t have an else clause. So let’s go ahead\nand get started with that.

\n

23 . 1If Statements

\n

This many chapters in, you know the drill. Any new feature starts in the front\nend and works its way through the pipeline. An if statement is, well, a\nstatement, so that’s where we hook it into the parser.

\n
  if (match(TOKEN_PRINT)) {\n    printStatement();\n
compiler.c
\nin statement()
\n
  } else if (match(TOKEN_IF)) {\n    ifStatement();\n
  } else if (match(TOKEN_LEFT_BRACE)) {\n
\n
compiler.c, in statement()
\n\n

When we see an if keyword, we hand off compilation to this function:

\n
compiler.c
\nadd after expressionStatement()
\n
static void ifStatement() {\n  consume(TOKEN_LEFT_PAREN, "Expect '(' after 'if'.");\n  expression();\n  consume(TOKEN_RIGHT_PAREN, "Expect ')' after condition."); \n\n  int thenJump = emitJump(OP_JUMP_IF_FALSE);\n  statement();\n\n  patchJump(thenJump);\n}\n
\n
compiler.c, add after expressionStatement()
\n\n\n

First we compile the condition expression, bracketed by parentheses. At runtime,\nthat will leave the condition value on top of the stack. We’ll use that to\ndetermine whether to execute the then branch or skip it.

\n

Then we emit a new OP_JUMP_IF_FALSE instruction. It has an operand for how\nmuch to offset the ip—how many bytes of code to skip. If the condition is\nfalsey, it adjusts the ip by that amount. Something like this:

\n\n

\"Flowchart\n

But we have a problem. When we’re writing the OP_JUMP_IF_FALSE instruction’s\noperand, how do we know how far to jump? We haven’t compiled the then branch\nyet, so we don’t know how much bytecode it contains.

\n

To fix that, we use a classic trick called backpatching. We emit the jump\ninstruction first with a placeholder offset operand. We keep track of where that\nhalf-finished instruction is. Next, we compile the then body. Once that’s done,\nwe know how far to jump. So we go back and replace that placeholder offset with\nthe real one now that we can calculate it. Sort of like sewing a patch onto the\nexisting fabric of the compiled code.

\"A\n

We encode this trick into two helper functions.

\n
compiler.c
\nadd after emitBytes()
\n
static int emitJump(uint8_t instruction) {\n  emitByte(instruction);\n  emitByte(0xff);\n  emitByte(0xff);\n  return currentChunk()->count - 2;\n}\n
\n
compiler.c, add after emitBytes()
\n\n

The first emits a bytecode instruction and writes a placeholder operand for the\njump offset. We pass in the opcode as an argument because later we’ll have two\ndifferent instructions that use this helper. We use two bytes for the jump\noffset operand. A 16-bit offset lets us jump over up\nto 65,535 bytes of code, which should be plenty for our needs.

\n\n

The function returns the offset of the emitted instruction in the chunk. After\ncompiling the then branch, we take that offset and pass it to this:

\n
compiler.c
\nadd after emitConstant()
\n
static void patchJump(int offset) {\n  // -2 to adjust for the bytecode for the jump offset itself.\n  int jump = currentChunk()->count - offset - 2;\n\n  if (jump > UINT16_MAX) {\n    error("Too much code to jump over.");\n  }\n\n  currentChunk()->code[offset] = (jump >> 8) & 0xff;\n  currentChunk()->code[offset + 1] = jump & 0xff;\n}\n
\n
compiler.c, add after emitConstant()
\n\n

This goes back into the bytecode and replaces the operand at the given location\nwith the calculated jump offset. We call patchJump() right before we emit the\nnext instruction that we want the jump to land on, so it uses the current\nbytecode count to determine how far to jump. In the case of an if statement,\nthat means right after we compile the then branch and before we compile the next\nstatement.

\n

That’s all we need at compile time. Let’s define the new instruction.

\n
  OP_PRINT,\n
chunk.h
\nin enum OpCode
\n
  OP_JUMP_IF_FALSE,\n
  OP_RETURN,\n
\n
chunk.h, in enum OpCode
\n\n

Over in the VM, we get it working like so:

\n
        break;\n      }\n
vm.c
\nin run()
\n
      case OP_JUMP_IF_FALSE: {\n        uint16_t offset = READ_SHORT();\n        if (isFalsey(peek(0))) vm.ip += offset;\n        break;\n      }\n
      case OP_RETURN: {\n
\n
vm.c, in run()
\n\n

This is the first instruction we’ve added that takes a 16-bit operand. To read\nthat from the chunk, we use a new macro.

\n
#define READ_CONSTANT() (vm.chunk->constants.values[READ_BYTE()])\n
vm.c
\nin run()
\n
#define READ_SHORT() \\\n    (vm.ip += 2, (uint16_t)((vm.ip[-2] << 8) | vm.ip[-1]))\n
#define READ_STRING() AS_STRING(READ_CONSTANT())\n
\n
vm.c, in run()
\n\n

It yanks the next two bytes from the chunk and builds a 16-bit unsigned integer\nout of them. As usual, we clean up our macro when we’re done with it.

\n
#undef READ_BYTE\n
vm.c
\nin run()
\n
#undef READ_SHORT\n
#undef READ_CONSTANT\n
\n
vm.c, in run()
\n\n

After reading the offset, we check the condition value on top of the stack.\nIf it’s falsey, we apply this jump offset to the ip.\nOtherwise, we leave the ip alone and execution will automatically proceed to\nthe next instruction following the jump instruction.

\n

In the case where the condition is falsey, we don’t need to do any other work.\nWe’ve offset the ip, so when the outer instruction dispatch loop turns again,\nit will pick up execution at that new instruction, past all of the code in the\nthen branch.

\n\n

Note that the jump instruction doesn’t pop the condition value off the stack. So\nwe aren’t totally done here, since this leaves an extra value floating around on\nthe stack. We’ll clean that up soon. Ignoring that for the moment, we do have a\nworking if statement in Lox now, with only one little instruction required to\nsupport it at runtime in the VM.

\n

23 . 1 . 1Else clauses

\n

An if statement without support for else clauses is like Morticia Addams\nwithout Gomez. So, after we compile the then branch, we look for an else\nkeyword. If we find one, we compile the else branch.

\n
  patchJump(thenJump);\n
compiler.c
\nin ifStatement()
\n
\n\n  if (match(TOKEN_ELSE)) statement();\n
}\n
\n
compiler.c, in ifStatement()
\n\n

When the condition is falsey, we’ll jump over the then branch. If there’s an\nelse branch, the ip will land right at the beginning of its code. But that’s\nnot enough, though. Here’s the flow that leads to:

\"Flowchart\n

If the condition is truthy, we execute the then branch like we want. But after\nthat, execution rolls right on through into the else branch. Oops! When the\ncondition is true, after we run the then branch, we need to jump over the else\nbranch. That way, in either case, we only execute a single branch, like this:

\"Flowchart\n

To implement that, we need another jump from the end of the then branch.

\n
  statement();\n\n
compiler.c
\nin ifStatement()
\n
  int elseJump = emitJump(OP_JUMP);\n\n
  patchJump(thenJump);\n
\n
compiler.c, in ifStatement()
\n\n

We patch that offset after the end of the else body.

\n
  if (match(TOKEN_ELSE)) statement();\n
compiler.c
\nin ifStatement()
\n
  patchJump(elseJump);\n
}\n
\n
compiler.c, in ifStatement()
\n\n

After executing the then branch, this jumps to the next statement after the else\nbranch. Unlike the other jump, this jump is unconditional. We always take it, so\nwe need another instruction that expresses that.

\n
  OP_PRINT,\n
chunk.h
\nin enum OpCode
\n
  OP_JUMP,\n
  OP_JUMP_IF_FALSE,\n
\n
chunk.h, in enum OpCode
\n\n

We interpret it like so:

\n
        break;\n      }\n
vm.c
\nin run()
\n
      case OP_JUMP: {\n        uint16_t offset = READ_SHORT();\n        vm.ip += offset;\n        break;\n      }\n
      case OP_JUMP_IF_FALSE: {\n
\n
vm.c, in run()
\n\n

Nothing too surprising here—the only difference is that it doesn’t check a\ncondition and always applies the offset.

\n

We have then and else branches working now, so we’re close. The last bit is to\nclean up that condition value we left on the stack. Remember, each statement is\nrequired to have zero stack effect—after the statement is finished executing,\nthe stack should be as tall as it was before.

\n

We could have the OP_JUMP_IF_FALSE instruction pop the condition itself, but\nsoon we’ll use that same instruction for the logical operators where we don’t\nwant the condition popped. Instead, we’ll have the compiler emit a couple of\nexplicit OP_POP instructions when compiling an if statement. We need to take\ncare that every execution path through the generated code pops the condition.

\n

When the condition is truthy, we pop it right before the code inside the then\nbranch.

\n
  int thenJump = emitJump(OP_JUMP_IF_FALSE);\n
compiler.c
\nin ifStatement()
\n
  emitByte(OP_POP);\n
  statement();\n
\n
compiler.c, in ifStatement()
\n\n

Otherwise, we pop it at the beginning of the else branch.

\n
  patchJump(thenJump);\n
compiler.c
\nin ifStatement()
\n
  emitByte(OP_POP);\n
\n\n  if (match(TOKEN_ELSE)) statement();\n
\n
compiler.c, in ifStatement()
\n\n

This little instruction here also means that every if statement has an\nimplicit else branch even if the user didn’t write an else clause. In the case\nwhere they left it off, all the branch does is discard the condition value.

\n

The full correct flow looks like this:

\"Flowchart\n

If you trace through, you can see that it always executes a single branch and\nensures the condition is popped first. All that remains is a little disassembler\nsupport.

\n
      return simpleInstruction("OP_PRINT", offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_JUMP:\n      return jumpInstruction("OP_JUMP", 1, chunk, offset);\n    case OP_JUMP_IF_FALSE:\n      return jumpInstruction("OP_JUMP_IF_FALSE", 1, chunk, offset);\n
    case OP_RETURN:\n
\n
debug.c, in disassembleInstruction()
\n\n

These two instructions have a new format with a 16-bit operand, so we add a new\nutility function to disassemble them.

\n
debug.c
\nadd after byteInstruction()
\n
static int jumpInstruction(const char* name, int sign,\n                           Chunk* chunk, int offset) {\n  uint16_t jump = (uint16_t)(chunk->code[offset + 1] << 8);\n  jump |= chunk->code[offset + 2];\n  printf("%-16s %4d -> %d\\n", name, offset,\n         offset + 3 + sign * jump);\n  return offset + 3;\n}\n
\n
debug.c, add after byteInstruction()
\n\n

There we go, that’s one complete control flow construct. If this were an ’80s\nmovie, the montage music would kick in and the rest of the control flow syntax\nwould take care of itself. Alas, the ’80s are long over,\nso we’ll have to grind it out ourselves.

\n\n

23 . 2Logical Operators

\n

You probably remember this from jlox, but the logical operators and and or\naren’t just another pair of binary operators like + and -. Because they\nshort-circuit and may not evaluate their right operand depending on the value of\nthe left one, they work more like control flow expressions.

\n

They’re basically a little variation on an if statement with an else clause.\nThe easiest way to explain them is to just show you the compiler code and the\ncontrol flow it produces in the resulting bytecode. Starting with and, we hook\nit into the expression parsing table here:

\n
  [TOKEN_NUMBER]        = {number,   NULL,   PREC_NONE},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_AND]           = {NULL,     and_,   PREC_AND},\n
  [TOKEN_CLASS]         = {NULL,     NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n

That hands off to a new parser function.

\n
compiler.c
\nadd after defineVariable()
\n
static void and_(bool canAssign) {\n  int endJump = emitJump(OP_JUMP_IF_FALSE);\n\n  emitByte(OP_POP);\n  parsePrecedence(PREC_AND);\n\n  patchJump(endJump);\n}\n
\n
compiler.c, add after defineVariable()
\n\n

At the point this is called, the left-hand side expression has already been\ncompiled. That means at runtime, its value will be on top of the stack. If that\nvalue is falsey, then we know the entire and must be false, so we skip the\nright operand and leave the left-hand side value as the result of the entire\nexpression. Otherwise, we discard the left-hand value and evaluate the right\noperand which becomes the result of the whole and expression.

\n

Those four lines of code right there produce exactly that. The flow looks like\nthis:

\"Flowchart\n

Now you can see why OP_JUMP_IF_FALSE leaves the\nvalue on top of the stack. When the left-hand side of the and is falsey, that\nvalue sticks around to become the result of the entire expression.

\n\n

23 . 2 . 1Logical or operator

\n

The or operator is a little more complex. First we add it to the parse table.

\n
  [TOKEN_NIL]           = {literal,  NULL,   PREC_NONE},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_OR]            = {NULL,     or_,    PREC_OR},\n
  [TOKEN_PRINT]         = {NULL,     NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n

When that parser consumes an infix or token, it calls this:

\n
compiler.c
\nadd after number()
\n
static void or_(bool canAssign) {\n  int elseJump = emitJump(OP_JUMP_IF_FALSE);\n  int endJump = emitJump(OP_JUMP);\n\n  patchJump(elseJump);\n  emitByte(OP_POP);\n\n  parsePrecedence(PREC_OR);\n  patchJump(endJump);\n}\n
\n
compiler.c, add after number()
\n\n

In an or expression, if the left-hand side is truthy, then we skip over the\nright operand. Thus we need to jump when a value is truthy. We could add a\nseparate instruction, but just to show how our compiler is free to map the\nlanguage’s semantics to whatever instruction sequence it wants, I implemented it\nin terms of the jump instructions we already have.

\n

When the left-hand side is falsey, it does a tiny jump over the next statement.\nThat statement is an unconditional jump over the code for the right operand.\nThis little dance effectively does a jump when the value is truthy. The flow\nlooks like this:

\"Flowchart\n

If I’m honest with you, this isn’t the best way to do this. There are more\ninstructions to dispatch and more overhead. There’s no good reason why or\nshould be slower than and. But it is kind of fun to see that it’s possible to\nimplement both operators without adding any new instructions. Forgive me my\nindulgences.

\n

OK, those are the three branching constructs in Lox. By that, I mean, these\nare the control flow features that only jump forward over code. Other\nlanguages often have some kind of multi-way branching statement like switch\nand maybe a conditional expression like ?:, but Lox keeps it simple.

\n

23 . 3While Statements

\n

That takes us to the looping statements, which jump backward so that code\ncan be executed more than once. Lox only has two loop constructs, while and\nfor. A while loop is (much) simpler, so we start the party there.

\n
    ifStatement();\n
compiler.c
\nin statement()
\n
  } else if (match(TOKEN_WHILE)) {\n    whileStatement();\n
  } else if (match(TOKEN_LEFT_BRACE)) {\n
\n
compiler.c, in statement()
\n\n

When we reach a while token, we call:

\n
compiler.c
\nadd after printStatement()
\n
static void whileStatement() {\n  consume(TOKEN_LEFT_PAREN, "Expect '(' after 'while'.");\n  expression();\n  consume(TOKEN_RIGHT_PAREN, "Expect ')' after condition.");\n\n  int exitJump = emitJump(OP_JUMP_IF_FALSE);\n  emitByte(OP_POP);\n  statement();\n\n  patchJump(exitJump);\n  emitByte(OP_POP);\n}\n
\n
compiler.c, add after printStatement()
\n\n

Most of this mirrors if statements—we compile the condition expression,\nsurrounded by mandatory parentheses. That’s followed by a jump instruction that\nskips over the subsequent body statement if the condition is falsey.

\n

We patch the jump after compiling the body and take care to pop the condition value from the stack on either path. The\nonly difference from an if statement is the loop. That looks like this:

\n\n
  statement();\n
compiler.c
\nin whileStatement()
\n
  emitLoop(loopStart);\n
\n\n  patchJump(exitJump);\n
\n
compiler.c, in whileStatement()
\n\n

After the body, we call this function to emit a “loop” instruction. That\ninstruction needs to know how far back to jump. When jumping forward, we had to\nemit the instruction in two stages since we didn’t know how far we were going to\njump until after we emitted the jump instruction. We don’t have that problem\nnow. We’ve already compiled the point in code that we want to jump back to—it’s right before the condition expression.

\n

All we need to do is capture that location as we compile it.

\n
static void whileStatement() {\n
compiler.c
\nin whileStatement()
\n
  int loopStart = currentChunk()->count;\n
  consume(TOKEN_LEFT_PAREN, "Expect '(' after 'while'.");\n
\n
compiler.c, in whileStatement()
\n\n

After executing the body of a while loop, we jump all the way back to before\nthe condition. That way, we re-evaluate the condition expression on each\niteration. We store the chunk’s current instruction count in loopStart to\nrecord the offset in the bytecode right before the condition expression we’re\nabout to compile. Then we pass that into this helper function:

\n
compiler.c
\nadd after emitBytes()
\n
static void emitLoop(int loopStart) {\n  emitByte(OP_LOOP);\n\n  int offset = currentChunk()->count - loopStart + 2;\n  if (offset > UINT16_MAX) error("Loop body too large.");\n\n  emitByte((offset >> 8) & 0xff);\n  emitByte(offset & 0xff);\n}\n
\n
compiler.c, add after emitBytes()
\n\n

It’s a bit like emitJump() and patchJump() combined. It emits a new loop\ninstruction, which unconditionally jumps backwards by a given offset. Like the\njump instructions, after that we have a 16-bit operand. We calculate the offset\nfrom the instruction we’re currently at to the loopStart point that we want to\njump back to. The + 2 is to take into account the size of the OP_LOOP\ninstruction’s own operands which we also need to jump over.

\n

From the VM’s perspective, there really is no semantic difference between\nOP_LOOP and OP_JUMP. Both just add an offset to the ip. We could have used\na single instruction for both and given it a signed offset operand. But I\nfigured it was a little easier to sidestep the annoying bit twiddling required\nto manually pack a signed 16-bit integer into two bytes, and we’ve got the\nopcode space available, so why not use it?

\n

The new instruction is here:

\n
  OP_JUMP_IF_FALSE,\n
chunk.h
\nin enum OpCode
\n
  OP_LOOP,\n
  OP_RETURN,\n
\n
chunk.h, in enum OpCode
\n\n

And in the VM, we implement it thusly:

\n
      }\n
vm.c
\nin run()
\n
      case OP_LOOP: {\n        uint16_t offset = READ_SHORT();\n        vm.ip -= offset;\n        break;\n      }\n
      case OP_RETURN: {\n
\n
vm.c, in run()
\n\n

The only difference from OP_JUMP is a subtraction instead of an addition.\nDisassembly is similar too.

\n
      return jumpInstruction("OP_JUMP_IF_FALSE", 1, chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_LOOP:\n      return jumpInstruction("OP_LOOP", -1, chunk, offset);\n
    case OP_RETURN:\n
\n
debug.c, in disassembleInstruction()
\n\n

That’s our while statement. It contains two jumps—a conditional forward one\nto escape the loop when the condition is not met, and an unconditional loop\nbackward after we have executed the body. The flow looks like this:

\"Flowchart\n

23 . 4For Statements

\n

The other looping statement in Lox is the venerable for loop, inherited from\nC. It’s got a lot more going on with it compared to a while loop. It has three\nclauses, all of which are optional:

\n

\n
    \n
  • \n

    The initializer can be a variable declaration or an expression. It runs once\nat the beginning of the statement.

    \n
    • \n
  • \n

    The condition clause is an expression. Like in a while loop, we exit the\nloop when it evaluates to something falsey.

    \n
    • \n
  • \n

    The increment expression runs once at the end of each loop iteration.

    \n
    • \n
\n\n

In jlox, the parser desugared a for loop to a synthesized AST for a while\nloop with some extra stuff before it and at the end of the body. We’ll do\nsomething similar, though we won’t go through anything like an AST. Instead,\nour bytecode compiler will use the jump and loop instructions we already have.

\n

We’ll work our way through the implementation a piece at a time, starting with\nthe for keyword.

\n
    printStatement();\n
compiler.c
\nin statement()
\n
  } else if (match(TOKEN_FOR)) {\n    forStatement();\n
  } else if (match(TOKEN_IF)) {\n
\n
compiler.c, in statement()
\n\n

It calls a helper function. If we only supported for loops with empty clauses\nlike for (;;), then we could implement it like this:

\n
compiler.c
\nadd after expressionStatement()
\n
static void forStatement() {\n  consume(TOKEN_LEFT_PAREN, "Expect '(' after 'for'.");\n  consume(TOKEN_SEMICOLON, "Expect ';'.");\n\n  int loopStart = currentChunk()->count;\n  consume(TOKEN_SEMICOLON, "Expect ';'.");\n  consume(TOKEN_RIGHT_PAREN, "Expect ')' after for clauses.");\n\n  statement();\n  emitLoop(loopStart);\n}\n
\n
compiler.c, add after expressionStatement()
\n\n

There’s a bunch of mandatory punctuation at the top. Then we compile the body.\nLike we did for while loops, we record the bytecode offset at the top of the\nbody and emit a loop to jump back to that point after it. We’ve got a working\nimplementation of infinite loops now.

\n\n

23 . 4 . 1Initializer clause

\n

Now we’ll add the first clause, the initializer. It executes only once, before\nthe body, so compiling is straightforward.

\n
  consume(TOKEN_LEFT_PAREN, "Expect '(' after 'for'.");\n
compiler.c
\nin forStatement()
\nreplace 1 line
\n
  if (match(TOKEN_SEMICOLON)) {\n    // No initializer.\n  } else if (match(TOKEN_VAR)) {\n    varDeclaration();\n  } else {\n    expressionStatement();\n  }\n
\n\n  int loopStart = currentChunk()->count;\n
\n
compiler.c, in forStatement(), replace 1 line
\n\n

The syntax is a little complex since we allow either a variable declaration or\nan expression. We use the presence of the var keyword to tell which we have.\nFor the expression case, we call expressionStatement() instead of\nexpression(). That looks for a semicolon, which we need here too, and also\nemits an OP_POP instruction to discard the value. We don’t want the\ninitializer to leave anything on the stack.

\n

If a for statement declares a variable, that variable should be scoped to the\nloop body. We ensure that by wrapping the whole statement in a scope.

\n
static void forStatement() {\n
compiler.c
\nin forStatement()
\n
  beginScope();\n
  consume(TOKEN_LEFT_PAREN, "Expect '(' after 'for'.");\n
\n
compiler.c, in forStatement()
\n\n

Then we close it at the end.

\n
  emitLoop(loopStart);\n
compiler.c
\nin forStatement()
\n
  endScope();\n
}\n
\n
compiler.c, in forStatement()
\n\n

23 . 4 . 2Condition clause

\n

Next, is the condition expression that can be used to exit the loop.

\n
  int loopStart = currentChunk()->count;\n
compiler.c
\nin forStatement()
\nreplace 1 line
\n
  int exitJump = -1;\n  if (!match(TOKEN_SEMICOLON)) {\n    expression();\n    consume(TOKEN_SEMICOLON, "Expect ';' after loop condition.");\n\n    // Jump out of the loop if the condition is false.\n    exitJump = emitJump(OP_JUMP_IF_FALSE);\n    emitByte(OP_POP); // Condition.\n  }\n\n
  consume(TOKEN_RIGHT_PAREN, "Expect ')' after for clauses.");\n
\n
compiler.c, in forStatement(), replace 1 line
\n\n

Since the clause is optional, we need to see if it’s actually present. If the\nclause is omitted, the next token must be a semicolon, so we look for that to\ntell. If there isn’t a semicolon, there must be a condition expression.

\n

In that case, we compile it. Then, just like with while, we emit a conditional\njump that exits the loop if the condition is falsey. Since the jump leaves the\nvalue on the stack, we pop it before executing the body. That ensures we discard\nthe value when the condition is true.

\n

After the loop body, we need to patch that jump.

\n
  emitLoop(loopStart);\n
compiler.c
\nin forStatement()
\n
\n\n  if (exitJump != -1) {\n    patchJump(exitJump);\n    emitByte(OP_POP); // Condition.\n  }\n\n
  endScope();\n}\n
\n
compiler.c, in forStatement()
\n\n

We do this only when there is a condition clause. If there isn’t, there’s no\njump to patch and no condition value on the stack to pop.

\n

23 . 4 . 3Increment clause

\n

I’ve saved the best for last, the increment clause. It’s pretty convoluted. It\nappears textually before the body, but executes after it. If we parsed to an\nAST and generated code in a separate pass, we could simply traverse into and\ncompile the for statement AST’s body field before its increment clause.

\n

Unfortunately, we can’t compile the increment clause later, since our compiler\nonly makes a single pass over the code. Instead, we’ll jump over the\nincrement, run the body, jump back up to the increment, run it, and then go to\nthe next iteration.

\n

I know, a little weird, but hey, it beats manually managing ASTs in memory in C,\nright? Here’s the code:

\n
  }\n\n
compiler.c
\nin forStatement()
\nreplace 1 line
\n
  if (!match(TOKEN_RIGHT_PAREN)) {\n    int bodyJump = emitJump(OP_JUMP);\n    int incrementStart = currentChunk()->count;\n    expression();\n    emitByte(OP_POP);\n    consume(TOKEN_RIGHT_PAREN, "Expect ')' after for clauses.");\n\n    emitLoop(loopStart);\n    loopStart = incrementStart;\n    patchJump(bodyJump);\n  }\n
\n\n  statement();\n
\n
compiler.c, in forStatement(), replace 1 line
\n\n

Again, it’s optional. Since this is the last clause, when omitted, the next\ntoken will be the closing parenthesis. When an increment is present, we need to\ncompile it now, but it shouldn’t execute yet. So, first, we emit an\nunconditional jump that hops over the increment clause’s code to the body of the\nloop.

\n

Next, we compile the increment expression itself. This is usually an assignment.\nWhatever it is, we only execute it for its side effect, so we also emit a pop to\ndiscard its value.

\n

The last part is a little tricky. First, we emit a loop instruction. This is the\nmain loop that takes us back to the top of the for loop—right before the\ncondition expression if there is one. That loop happens right after the\nincrement, since the increment executes at the end of each loop iteration.

\n

Then we change loopStart to point to the offset where the increment expression\nbegins. Later, when we emit the loop instruction after the body statement, this\nwill cause it to jump up to the increment expression instead of the top of the\nloop like it does when there is no increment. This is how we weave the\nincrement in to run after the body.

\n

It’s convoluted, but it all works out. A complete loop with all the clauses\ncompiles to a flow like this:

\"Flowchart\n

As with implementing for loops in jlox, we didn’t need to touch the runtime.\nIt all gets compiled down to primitive control flow operations the VM already\nsupports. In this chapter, we’ve taken a big leap\nforward—clox is now Turing complete. We’ve also covered quite a bit of new\nsyntax: three statements and two expression forms. Even so, it only took three\nnew simple instructions. That’s a pretty good effort-to-reward ratio for the\narchitecture of our VM.

\n\n
\n

Challenges

\n
    \n
  1. \n

    In addition to if statements, most C-family languages have a multi-way\nswitch statement. Add one to clox. The grammar is:

    \n
    switchStmt     → "switch" "(" expression ")"\n                 "{" switchCase* defaultCase? "}" ;\nswitchCase     → "case" expression ":" statement* ;\ndefaultCase    → "default" ":" statement* ;\n
    \n

    To execute a switch statement, first evaluate the parenthesized switch\nvalue expression. Then walk the cases. For each case, evaluate its value\nexpression. If the case value is equal to the switch value, execute the\nstatements under the case and then exit the switch statement. Otherwise,\ntry the next case. If no case matches and there is a default clause,\nexecute its statements.

    \n

    To keep things simpler, we’re omitting fallthrough and break statements.\nEach case automatically jumps to the end of the switch statement after its\nstatements are done.

    \n
    2. \n
  2. \n

    In jlox, we had a challenge to add support for break statements. This\ntime, let’s do continue:

    \n
    continueStmt   → "continue" ";" ;\n
    \n

    A continue statement jumps directly to the top of the nearest enclosing\nloop, skipping the rest of the loop body. Inside a for loop, a continue\njumps to the increment clause, if there is one. It’s a compile-time error to\nhave a continue statement not enclosed in a loop.

    \n

    Make sure to think about scope. What should happen to local variables\ndeclared inside the body of the loop or in blocks nested inside the loop\nwhen a continue is executed?

    \n
    3. \n
  3. \n

    Control flow constructs have been mostly unchanged since Algol 68. Language\nevolution since then has focused on making code more declarative and high\nlevel, so imperative control flow hasn’t gotten much attention.

    \n

    For fun, try to invent a useful novel control flow feature for Lox. It can\nbe a refinement of an existing form or something entirely new. In practice,\nit’s hard to come up with something useful enough at this low expressiveness\nlevel to outweigh the cost of forcing a user to learn an unfamiliar notation\nand behavior, but it’s a good chance to practice your design skills.

    \n
    4. \n
\n
\n
\n

Design Note: Considering Goto Harmful

\n

Discovering that all of our beautiful structured control flow in Lox is actually\ncompiled to raw unstructured jumps is like the moment in Scooby Doo when the\nmonster rips the mask off their face. It was goto all along! Except in this\ncase, the monster is under the mask. We all know goto is evil. But . . . why?

\n

It is true that you can write outrageously unmaintainable code using goto. But I\ndon’t think most programmers around today have seen that first hand. It’s been a\nlong time since that style was common. These days, it’s a boogie man we invoke\nin scary stories around the campfire.

\n

The reason we rarely confront that monster in person is because Edsger Dijkstra\nslayed it with his famous letter “Go To Statement Considered Harmful”, published\nin Communications of the ACM (March, 1968). Debate around structured\nprogramming had been fierce for some time with adherents on both sides, but I\nthink Dijkstra deserves the most credit for effectively ending it. Most new\nlanguages today have no unstructured jump statements.

\n

A one-and-a-half page letter that almost single-handedly destroyed a language\nfeature must be pretty impressive stuff. If you haven’t read it, I encourage you\nto do so. It’s a seminal piece of computer science lore, one of our tribe’s\nancestral songs. Also, it’s a nice, short bit of practice for reading academic\nCS writing, which is a useful skill to develop.

\n\n

I’ve read it through a number of times, along with a few critiques, responses,\nand commentaries. I ended up with mixed feelings, at best. At a very high level,\nI’m with him. His general argument is something like this:

\n
    \n
  1. \n

    As programmers, we write programs—static text—but what we care about\nis the actual running program—its dynamic behavior.

    \n
    2. \n
  2. \n

    We’re better at reasoning about static things than dynamic things. (He\ndoesn’t provide any evidence to support this claim, but I accept it.)

    \n
    3. \n
  3. \n

    Thus, the more we can make the dynamic execution of the program reflect its\ntextual structure, the better.

    \n
    4. \n
\n

This is a good start. Drawing our attention to the separation between the code\nwe write and the code as it runs inside the machine is an interesting insight.\nThen he tries to define a “correspondence” between program text and execution.\nFor someone who spent literally his entire career advocating greater rigor in\nprogramming, his definition is pretty hand-wavey. He says:

\n
\n

Let us now consider how we can characterize the progress of a process. (You\nmay think about this question in a very concrete manner: suppose that a\nprocess, considered as a time succession of actions, is stopped after an\narbitrary action, what data do we have to fix in order that we can redo the\nprocess until the very same point?)

\n
\n

Imagine it like this. You have two computers with the same program running on\nthe exact same inputs—so totally deterministic. You pause one of them at an\narbitrary point in its execution. What data would you need to send to the other\ncomputer to be able to stop it exactly as far along as the first one was?

\n

If your program allows only simple statements like assignment, it’s easy. You\njust need to know the point after the last statement you executed. Basically a\nbreakpoint, the ip in our VM, or the line number in an error message. Adding\nbranching control flow like if and switch doesn’t add any more to this. Even\nif the marker points inside a branch, we can still tell where we are.

\n

Once you add function calls, you need something more. You could have paused the\nfirst computer in the middle of a function, but that function may be called from\nmultiple places. To pause the second machine at exactly the same point in the\nentire program’s execution, you need to pause it on the right call to that\nfunction.

\n

So you need to know not just the current statement, but, for function calls that\nhaven’t returned yet, you need to know the locations of the callsites. In other\nwords, a call stack, though I don’t think that term existed when Dijkstra wrote\nthis. Groovy.

\n

He notes that loops make things harder. If you pause in the middle of a loop\nbody, you don’t know how many iterations have run. So he says you also need to\nkeep an iteration count. And, since loops can nest, you need a stack of those\n(presumably interleaved with the call stack pointers since you can be in loops\nin outer calls too).

\n

This is where it gets weird. So we’re really building to something now, and you\nexpect him to explain how goto breaks all of this. Instead, he just says:

\n
\n

The unbridled use of the go to statement has an immediate consequence that it\nbecomes terribly hard to find a meaningful set of coordinates in which to\ndescribe the process progress.

\n
\n

He doesn’t prove that this is hard, or say why. He just says it. He does say\nthat one approach is unsatisfactory:

\n
\n

With the go to statement one can, of course, still describe the progress\nuniquely by a counter counting the number of actions performed since program\nstart (viz. a kind of normalized clock). The difficulty is that such a\ncoordinate, although unique, is utterly unhelpful.

\n
\n

But . . . that’s effectively what loop counters do, and he was fine with those.\nIt’s not like every loop is a simple “for every integer from 0 to 10”\nincrementing count. Many are while loops with complex conditionals.

\n

Taking an example close to home, consider the core bytecode execution loop at\nthe heart of clox. Dijkstra argues that that loop is tractable because we can\nsimply count how many times the loop has run to reason about its progress. But\nthat loop runs once for each executed instruction in some user’s compiled Lox\nprogram. Does knowing that it executed 6,201 bytecode instructions really tell\nus VM maintainers anything edifying about the state of the interpreter?

\n

In fact, this particular example points to a deeper truth. Böhm and Jacopini\nproved that any control flow using goto can be transformed into one using\njust sequencing, loops, and branches. Our bytecode interpreter loop is a living\nexample of that proof: it implements the unstructured control flow of the clox\nbytecode instruction set without using any gotos itself.

\n

That seems to offer a counter-argument to Dijkstra’s claim: you can define a\ncorrespondence for a program using gotos by transforming it to one that doesn’t\nand then use the correspondence from that program, which—according to him—is acceptable because it uses only branches and loops.

\n

But, honestly, my argument here is also weak. I think both of us are basically\ndoing pretend math and using fake logic to make what should be an empirical,\nhuman-centered argument. Dijkstra is right that some code using goto is really\nbad. Much of that could and should be turned into clearer code by using\nstructured control flow.

\n

By eliminating goto completely from languages, you’re definitely prevented from\nwriting bad code using gotos. It may be that forcing users to use structured\ncontrol flow and making it an uphill battle to write goto-like code using those\nconstructs is a net win for all of our productivity.

\n

But I do wonder sometimes if we threw out the baby with the bathwater. In the\nabsence of goto, we often resort to more complex structured patterns. The\n“switch inside a loop” is a classic one. Another is using a guard variable to\nexit out of a series of nested loops:

\n\n
// See if the matrix contains a zero.\nbool found = false;\nfor (int x = 0; x < xSize; x++) {\n  for (int y = 0; y < ySize; y++) {\n    for (int z = 0; z < zSize; z++) {\n      if (matrix[x][y][z] == 0) {\n        printf("found");\n        found = true;\n        break;\n      }\n    }\n    if (found) break;\n  }\n  if (found) break;\n}\n
\n

Is that really better than:

\n
for (int x = 0; x < xSize; x++) {\n  for (int y = 0; y < ySize; y++) {\n    for (int z = 0; z < zSize; z++) {\n      if (matrix[x][y][z] == 0) {\n        printf("found");\n        goto done;\n      }\n    }\n  }\n}\ndone:\n
\n\n

I guess what I really don’t like is that we’re making language design and\nengineering decisions today based on fear. Few people today have any subtle\nunderstanding of the problems and benefits of goto. Instead, we just think it’s\n“considered harmful”. Personally, I’ve never found dogma a good starting place\nfor quality creative work.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/local-variables.html", "content": "\n\n\n\nLocal Variables · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
22
\n

Local Variables

\n\n
\n

And as imagination bodies forth
\nThe forms of things unknown, the poet’s pen
\nTurns them to shapes and gives to airy nothing
\nA local habitation and a name.

\n

William Shakespeare, A Midsummer Night’s Dream

\n
\n

The last chapter introduced variables to clox, but only of the global variety. In this chapter, we’ll extend that to\nsupport blocks, block scope, and local variables. In jlox, we managed to pack\nall of that and globals into one chapter. For clox, that’s two chapters worth of\nwork partially because, frankly, everything takes more effort in C.

\n\n

But an even more important reason is that our approach to local variables will\nbe quite different from how we implemented globals. Global variables are late\nbound in Lox. “Late” in this context means “resolved after compile time”. That’s\ngood for keeping the compiler simple, but not great for performance. Local\nvariables are one of the most-used parts of a\nlanguage. If locals are slow, everything is slow. So we want a strategy for\nlocal variables that’s as efficient as possible.

\n\n

Fortunately, lexical scoping is here to help us. As the name implies, lexical\nscope means we can resolve a local variable just by looking at the text of the\nprogram—locals are not late bound. Any processing work we do in the\ncompiler is work we don’t have to do at runtime, so our implementation of\nlocal variables will lean heavily on the compiler.

\n

22 . 1Representing Local Variables

\n

The nice thing about hacking on a programming language in modern times is\nthere’s a long lineage of other languages to learn from. So how do C and Java\nmanage their local variables? Why, on the stack, of course! They typically use\nthe native stack mechanisms supported by the chip and OS. That’s a little too\nlow level for us, but inside the virtual world of clox, we have our own stack we\ncan use.

\n

Right now, we only use it for holding on to temporaries—short-lived blobs\nof data that we need to remember while computing an expression. As long as we\ndon’t get in the way of those, we can stuff our local variables onto the stack\ntoo. This is great for performance. Allocating space for a new local requires\nonly incrementing the stackTop pointer, and freeing is likewise a decrement.\nAccessing a variable from a known stack slot is an indexed array lookup.

\n

We do need to be careful, though. The VM expects the stack to behave like, well,\na stack. We have to be OK with allocating new locals only on the top of the\nstack, and we have to accept that we can discard a local only when nothing is\nabove it on the stack. Also, we need to make sure temporaries don’t interfere.

\n

Conveniently, the design of Lox is in harmony with\nthese constraints. New locals are always created by declaration statements.\nStatements don’t nest inside expressions, so there are never any temporaries on\nthe stack when a statement begins executing. Blocks are strictly nested. When a\nblock ends, it always takes the innermost, most recently declared locals with\nit. Since those are also the locals that came into scope last, they should be on\ntop of the stack where we need them.

\n\n

Step through this example program and watch how the local variables come in and\ngo out of scope:

\"A\n

See how they fit a stack perfectly? It seems that the stack will work for\nstoring locals at runtime. But we can go further than that. Not only do we know\nthat they will be on the stack, but we can even pin down precisely where\nthey will be on the stack. Since the compiler knows exactly which local\nvariables are in scope at any point in time, it can effectively simulate the\nstack during compilation and note where in the stack each\nvariable lives.

\n

We’ll take advantage of this by using these stack offsets as operands for the\nbytecode instructions that read and store local variables. This makes working\nwith locals deliciously fast—as simple as indexing into an array.

\n\n

There’s a lot of state we need to track in the compiler to make this whole thing\ngo, so let’s get started there. In jlox, we used a linked chain of “environment”\nHashMaps to track which local variables were currently in scope. That’s sort of\nthe classic, schoolbook way of representing lexical scope. For clox, as usual,\nwe’re going a little closer to the metal. All of the state lives in a new\nstruct.

\n
} ParseRule;\n
compiler.c
\nadd after struct ParseRule
\n
\n\ntypedef struct {\n  Local locals[UINT8_COUNT];\n  int localCount;\n  int scopeDepth;\n} Compiler;\n
\n\nParser parser;\n
\n
compiler.c, add after struct ParseRule
\n\n

We have a simple, flat array of all locals that are in scope during each point in\nthe compilation process. They are ordered in the array\nin the order that their declarations appear in the code. Since the instruction\noperand we’ll use to encode a local is a single byte, our VM has a hard limit on\nthe number of locals that can be in scope at once. That means we can also give\nthe locals array a fixed size.

\n\n
#define DEBUG_TRACE_EXECUTION\n
common.h
\n
\n\n#define UINT8_COUNT (UINT8_MAX + 1)\n
\n\n#endif\n
\n
common.h
\n\n

Back in the Compiler struct, the localCount field tracks how many locals are\nin scope—how many of those array slots are in use. We also track the “scope\ndepth”. This is the number of blocks surrounding the current bit of code we’re\ncompiling.

\n

Our Java interpreter used a chain of maps to keep each block’s variables\nseparate from other blocks’. This time, we’ll simply number variables with the\nlevel of nesting where they appear. Zero is the global scope, one is the first\ntop-level block, two is inside that, you get the idea. We use this to track\nwhich block each local belongs to so that we know which locals to discard when a\nblock ends.

\n

Each local in the array is one of these:

\n
} ParseRule;\n
compiler.c
\nadd after struct ParseRule
\n
\n\ntypedef struct {\n  Token name;\n  int depth;\n} Local;\n
\n\ntypedef struct {\n
\n
compiler.c, add after struct ParseRule
\n\n

We store the name of the variable. When we’re resolving an identifier, we\ncompare the identifier’s lexeme with each local’s name to find a match. It’s\npretty hard to resolve a variable if you don’t know its name. The depth field\nrecords the scope depth of the block where the local variable was declared.\nThat’s all the state we need for now.

\n

This is a very different representation from what we had in jlox, but it still\nlets us answer all of the same questions our compiler needs to ask of the\nlexical environment. The next step is figuring out how the compiler gets at\nthis state. If we were principled engineers, we’d\ngive each function in the front end a parameter that accepts a pointer to a\nCompiler. We’d create a Compiler at the beginning and carefully thread it\nthrough each function call . . . but that would mean a lot of boring changes to\nthe code we already wrote, so here’s a global variable instead:

\n\n
Parser parser;\n
compiler.c
\nadd after variable parser
\n
Compiler* current = NULL;\n
Chunk* compilingChunk;\n
\n
compiler.c, add after variable parser
\n\n

Here’s a little function to initialize the compiler:

\n
compiler.c
\nadd after emitConstant()
\n
static void initCompiler(Compiler* compiler) {\n  compiler->localCount = 0;\n  compiler->scopeDepth = 0;\n  current = compiler;\n}\n
\n
compiler.c, add after emitConstant()
\n\n

When we first start up the VM, we call it to get everything into a clean state.

\n
  initScanner(source);\n
compiler.c
\nin compile()
\n
  Compiler compiler;\n  initCompiler(&compiler);\n
  compilingChunk = chunk;\n
\n
compiler.c, in compile()
\n\n

Our compiler has the data it needs, but not the operations on that data. There’s\nno way to create and destroy scopes, or add and resolve variables. We’ll add\nthose as we need them. First, let’s start building some language features.

\n

22 . 2Block Statements

\n

Before we can have any local variables, we need some local scopes. These come\nfrom two things: function bodies and blocks. Functions\nare a big chunk of work that we’ll tackle in a later chapter, so\nfor now we’re only going to do blocks. As usual, we start with the syntax. The\nnew grammar we’ll introduce is:

\n
statement      → exprStmt\n               | printStmt\n               | block ;\n\nblock          → "{" declaration* "}" ;\n
\n\n

Blocks are a kind of statement, so the rule for them goes in the statement\nproduction. The corresponding code to compile one looks like this:

\n
  if (match(TOKEN_PRINT)) {\n    printStatement();\n
compiler.c
\nin statement()
\n
  } else if (match(TOKEN_LEFT_BRACE)) {\n    beginScope();\n    block();\n    endScope();\n
  } else {\n
\n
compiler.c, in statement()
\n\n

After parsing the initial curly brace, we use this\nhelper function to compile the rest of the block:

\n\n
compiler.c
\nadd after expression()
\n
static void block() {\n  while (!check(TOKEN_RIGHT_BRACE) && !check(TOKEN_EOF)) {\n    declaration();\n  }\n\n  consume(TOKEN_RIGHT_BRACE, "Expect '}' after block.");\n}\n
\n
compiler.c, add after expression()
\n\n

It keeps parsing declarations and statements until it hits the closing brace. As\nwe do with any loop in the parser, we also check for the end of the token\nstream. This way, if there’s a malformed program with a missing closing curly,\nthe compiler doesn’t get stuck in a loop.

\n

Executing a block simply means executing the statements it contains, one after\nthe other, so there isn’t much to compiling them. The semantically interesting\nthing blocks do is create scopes. Before we compile the body of a block, we call\nthis function to enter a new local scope:

\n
compiler.c
\nadd after endCompiler()
\n
static void beginScope() {\n  current->scopeDepth++;\n}\n
\n
compiler.c, add after endCompiler()
\n\n

In order to “create” a scope, all we do is increment the current depth. This is\ncertainly much faster than jlox, which allocated an entire new HashMap for\neach one. Given beginScope(), you can probably guess what endScope() does.

\n
compiler.c
\nadd after beginScope()
\n
static void endScope() {\n  current->scopeDepth--;\n}\n
\n
compiler.c, add after beginScope()
\n\n

That’s it for blocks and scopes—more or less—so we’re ready to stuff some\nvariables into them.

\n

22 . 3Declaring Local Variables

\n

Usually we start with parsing here, but our compiler already supports parsing\nand compiling variable declarations. We’ve got var statements, identifier\nexpressions and assignment in there now. It’s just that the compiler assumes\nall variables are global. So we don’t need any new parsing support, we just need\nto hook up the new scoping semantics to the existing code.

\"The\n

Variable declaration parsing begins in varDeclaration() and relies on a couple\nof other functions. First, parseVariable() consumes the identifier token for\nthe variable name, adds its lexeme to the chunk’s constant table as a string,\nand then returns the constant table index where it was added. Then, after\nvarDeclaration() compiles the initializer, it calls defineVariable() to emit\nthe bytecode for storing the variable’s value in the global variable hash table.

\n

Both of those helpers need a few changes to support local variables. In\nparseVariable(), we add:

\n
  consume(TOKEN_IDENTIFIER, errorMessage);\n
compiler.c
\nin parseVariable()
\n
\n\n  declareVariable();\n  if (current->scopeDepth > 0) return 0;\n\n
  return identifierConstant(&parser.previous);\n
\n
compiler.c, in parseVariable()
\n\n

First, we “declare” the variable. I’ll get to what that means in a second. After\nthat, we exit the function if we’re in a local scope. At runtime, locals aren’t\nlooked up by name. There’s no need to stuff the variable’s name into the\nconstant table, so if the declaration is inside a local scope, we return a dummy\ntable index instead.

\n

Over in defineVariable(), we need to emit the code to store a local variable\nif we’re in a local scope. It looks like this:

\n
static void defineVariable(uint8_t global) {\n
compiler.c
\nin defineVariable()
\n
  if (current->scopeDepth > 0) {\n    return;\n  }\n\n
  emitBytes(OP_DEFINE_GLOBAL, global);\n
\n
compiler.c, in defineVariable()
\n\n

Wait, what? Yup. That’s it. There is no code to create a local variable at\nruntime. Think about what state the VM is in. It has already executed the code\nfor the variable’s initializer (or the implicit nil if the user omitted an\ninitializer), and that value is sitting right on top of the stack as the only\nremaining temporary. We also know that new locals are allocated at the top of\nthe stack . . . right where that value already is. Thus, there’s nothing to do. The\ntemporary simply becomes the local variable. It doesn’t get much more\nefficient than that.

\n

\"Walking\n\n

OK, so what’s “declaring” about? Here’s what that does:

\n
compiler.c
\nadd after identifierConstant()
\n
static void declareVariable() {\n  if (current->scopeDepth == 0) return;\n\n  Token* name = &parser.previous;\n  addLocal(*name);\n}\n
\n
compiler.c, add after identifierConstant()
\n\n

This is the point where the compiler records the existence of the variable. We\nonly do this for locals, so if we’re in the top-level global scope, we just bail\nout. Because global variables are late bound, the compiler doesn’t keep track of\nwhich declarations for them it has seen.

\n

But for local variables, the compiler does need to remember that the variable\nexists. That’s what declaring it does—it adds it to the compiler’s list of\nvariables in the current scope. We implement that using another new function.

\n
compiler.c
\nadd after identifierConstant()
\n
static void addLocal(Token name) {\n  Local* local = &current->locals[current->localCount++];\n  local->name = name;\n  local->depth = current->scopeDepth;\n}\n
\n
compiler.c, add after identifierConstant()
\n\n

This initializes the next available Local in the compiler’s array of variables.\nIt stores the variable’s name and the depth of the\nscope that owns the variable.

\n\n

Our implementation is fine for a correct Lox program, but what about invalid\ncode? Let’s aim to be robust. The first error to handle is not really the user’s\nfault, but more a limitation of the VM. The instructions to work with local\nvariables refer to them by slot index. That index is stored in a single-byte\noperand, which means the VM only supports up to 256 local variables in scope at\none time.

\n

If we try to go over that, not only could we not refer to them at runtime, but\nthe compiler would overwrite its own locals array, too. Let’s prevent that.

\n
static void addLocal(Token name) {\n
compiler.c
\nin addLocal()
\n
  if (current->localCount == UINT8_COUNT) {\n    error("Too many local variables in function.");\n    return;\n  }\n\n
  Local* local = &current->locals[current->localCount++];\n
\n
compiler.c, in addLocal()
\n\n

The next case is trickier. Consider:

\n
{\n  var a = "first";\n  var a = "second";\n}\n
\n

At the top level, Lox allows redeclaring a variable with the same name as a\nprevious declaration because that’s useful for the REPL. But inside a local\nscope, that’s a pretty weird thing to do. It’s likely\nto be a mistake, and many languages, including our own Lox, enshrine that\nassumption by making this an error.

\n\n

Note that the above program is different from this one:

\n
{\n  var a = "outer";\n  {\n    var a = "inner";\n  }\n}\n
\n

It’s OK to have two variables with the same name in different scopes, even\nwhen the scopes overlap such that both are visible at the same time. That’s\nshadowing, and Lox does allow that. It’s only an error to have two variables\nwith the same name in the same local scope.

\n

We detect that error like so:

\n
  Token* name = &parser.previous;\n
compiler.c
\nin declareVariable()
\n
  for (int i = current->localCount - 1; i >= 0; i--) {\n    Local* local = &current->locals[i];\n    if (local->depth != -1 && local->depth < current->scopeDepth) {\n      break; \n    }\n\n    if (identifiersEqual(name, &local->name)) {\n      error("Already a variable with this name in this scope.");\n    }\n  }\n\n
  addLocal(*name);\n}\n
\n
compiler.c, in declareVariable()
\n\n\n

Local variables are appended to the array when they’re declared, which means the\ncurrent scope is always at the end of the array. When we declare a new variable,\nwe start at the end and work backward, looking for an existing variable with the\nsame name. If we find one in the current scope, we report the error. Otherwise,\nif we reach the beginning of the array or a variable owned by another scope,\nthen we know we’ve checked all of the existing variables in the scope.

\n

To see if two identifiers are the same, we use this:

\n
compiler.c
\nadd after identifierConstant()
\n
static bool identifiersEqual(Token* a, Token* b) {\n  if (a->length != b->length) return false;\n  return memcmp(a->start, b->start, a->length) == 0;\n}\n
\n
compiler.c, add after identifierConstant()
\n\n

Since we know the lengths of both lexemes, we check that first. That will fail\nquickly for many non-equal strings. If the lengths are\nthe same, we check the characters using memcmp(). To get to memcmp(), we\nneed an include.

\n\n
#include <stdlib.h>\n
compiler.c
\n
#include <string.h>\n
\n\n#include "common.h"\n
\n
compiler.c
\n\n

With this, we’re able to bring variables into being. But, like ghosts, they\nlinger on beyond the scope where they are declared. When a block ends, we need\nto put them to rest.

\n
  current->scopeDepth--;\n
compiler.c
\nin endScope()
\n
\n\n  while (current->localCount > 0 &&\n         current->locals[current->localCount - 1].depth >\n            current->scopeDepth) {\n    emitByte(OP_POP);\n    current->localCount--;\n  }\n
}\n
\n
compiler.c, in endScope()
\n\n

When we pop a scope, we walk backward through the local array looking for any\nvariables declared at the scope depth we just left. We discard them by simply\ndecrementing the length of the array.

\n

There is a runtime component to this too. Local variables occupy slots on the\nstack. When a local variable goes out of scope, that slot is no longer needed\nand should be freed. So, for each variable that we discard, we also emit an\nOP_POP instruction to pop it from the stack.

\n\n

22 . 4Using Locals

\n

We can now compile and execute local variable declarations. At runtime, their\nvalues are sitting where they should be on the stack. Let’s start using them.\nWe’ll do both variable access and assignment at the same time since they touch\nthe same functions in the compiler.

\n

We already have code for getting and setting global variables, and—like good\nlittle software engineers—we want to reuse as much of that existing code as\nwe can. Something like this:

\n
static void namedVariable(Token name, bool canAssign) {\n
compiler.c
\nin namedVariable()
\nreplace 1 line
\n
  uint8_t getOp, setOp;\n  int arg = resolveLocal(current, &name);\n  if (arg != -1) {\n    getOp = OP_GET_LOCAL;\n    setOp = OP_SET_LOCAL;\n  } else {\n    arg = identifierConstant(&name);\n    getOp = OP_GET_GLOBAL;\n    setOp = OP_SET_GLOBAL;\n  }\n
\n\n  if (canAssign && match(TOKEN_EQUAL)) {\n
\n
compiler.c, in namedVariable(), replace 1 line
\n\n

Instead of hardcoding the bytecode instructions emitted for variable access and\nassignment, we use a couple of C variables. First, we try to find a local\nvariable with the given name. If we find one, we use the instructions for\nworking with locals. Otherwise, we assume it’s a global variable and use the\nexisting bytecode instructions for globals.

\n

A little further down, we use those variables to emit the right instructions.\nFor assignment:

\n
  if (canAssign && match(TOKEN_EQUAL)) {\n    expression();\n
compiler.c
\nin namedVariable()
\nreplace 1 line
\n
    emitBytes(setOp, (uint8_t)arg);\n
  } else {\n
\n
compiler.c, in namedVariable(), replace 1 line
\n\n

And for access:

\n
    emitBytes(setOp, (uint8_t)arg);\n  } else {\n
compiler.c
\nin namedVariable()
\nreplace 1 line
\n
    emitBytes(getOp, (uint8_t)arg);\n
  }\n
\n
compiler.c, in namedVariable(), replace 1 line
\n\n

The real heart of this chapter, the part where we resolve a local variable, is\nhere:

\n
compiler.c
\nadd after identifiersEqual()
\n
static int resolveLocal(Compiler* compiler, Token* name) {\n  for (int i = compiler->localCount - 1; i >= 0; i--) {\n    Local* local = &compiler->locals[i];\n    if (identifiersEqual(name, &local->name)) {\n      return i;\n    }\n  }\n\n  return -1;\n}\n
\n
compiler.c, add after identifiersEqual()
\n\n

For all that, it’s straightforward. We walk the list of locals that are\ncurrently in scope. If one has the same name as the identifier token, the\nidentifier must refer to that variable. We’ve found it! We walk the array\nbackward so that we find the last declared variable with the identifier. That\nensures that inner local variables correctly shadow locals with the same name in\nsurrounding scopes.

\n

At runtime, we load and store locals using the stack slot index, so that’s what\nthe compiler needs to calculate after it resolves the variable. Whenever a\nvariable is declared, we append it to the locals array in Compiler. That means\nthe first local variable is at index zero, the next one is at index one, and so\non. In other words, the locals array in the compiler has the exact same layout\nas the VM’s stack will have at runtime. The variable’s index in the locals array\nis the same as its stack slot. How convenient!

\n

If we make it through the whole array without finding a variable with the given\nname, it must not be a local. In that case, we return -1 to signal that it\nwasn’t found and should be assumed to be a global variable instead.

\n

22 . 4 . 1Interpreting local variables

\n

Our compiler is emitting two new instructions, so let’s get them working. First\nis loading a local variable:

\n
  OP_POP,\n
chunk.h
\nin enum OpCode
\n
  OP_GET_LOCAL,\n
  OP_GET_GLOBAL,\n
\n
chunk.h, in enum OpCode
\n\n

And its implementation:

\n
      case OP_POP: pop(); break;\n
vm.c
\nin run()
\n
      case OP_GET_LOCAL: {\n        uint8_t slot = READ_BYTE();\n        push(vm.stack[slot]); \n        break;\n      }\n
      case OP_GET_GLOBAL: {\n
\n
vm.c, in run()
\n\n

It takes a single-byte operand for the stack slot where the local lives. It\nloads the value from that index and then pushes it on top of the stack where\nlater instructions can find it.

\n\n

Next is assignment:

\n
  OP_GET_LOCAL,\n
chunk.h
\nin enum OpCode
\n
  OP_SET_LOCAL,\n
  OP_GET_GLOBAL,\n
\n
chunk.h, in enum OpCode
\n\n

You can probably predict the implementation.

\n
      }\n
vm.c
\nin run()
\n
      case OP_SET_LOCAL: {\n        uint8_t slot = READ_BYTE();\n        vm.stack[slot] = peek(0);\n        break;\n      }\n
      case OP_GET_GLOBAL: {\n
\n
vm.c, in run()
\n\n

It takes the assigned value from the top of the stack and stores it in the stack\nslot corresponding to the local variable. Note that it doesn’t pop the value\nfrom the stack. Remember, assignment is an expression, and every expression\nproduces a value. The value of an assignment expression is the assigned value\nitself, so the VM just leaves the value on the stack.

\n

Our disassembler is incomplete without support for these two new instructions.

\n
      return simpleInstruction("OP_POP", offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_GET_LOCAL:\n      return byteInstruction("OP_GET_LOCAL", chunk, offset);\n    case OP_SET_LOCAL:\n      return byteInstruction("OP_SET_LOCAL", chunk, offset);\n
    case OP_GET_GLOBAL:\n
\n
debug.c, in disassembleInstruction()
\n\n

The compiler compiles local variables to direct slot access. The local\nvariable’s name never leaves the compiler to make it into the chunk at all.\nThat’s great for performance, but not so great for introspection. When we\ndisassemble these instructions, we can’t show the variable’s name like we could\nwith globals. Instead, we just show the slot number.

\n\n
debug.c
\nadd after simpleInstruction()
\n
static int byteInstruction(const char* name, Chunk* chunk,\n                           int offset) {\n  uint8_t slot = chunk->code[offset + 1];\n  printf("%-16s %4d\\n", name, slot);\n  return offset + 2; \n}\n
\n
debug.c, add after simpleInstruction()
\n\n

22 . 4 . 2Another scope edge case

\n

We already sunk some time into handling a couple of weird edge cases around\nscopes. We made sure shadowing works correctly. We report an error if two\nvariables in the same local scope have the same name. For reasons that aren’t\nentirely clear to me, variable scoping seems to have a lot of these wrinkles.\nI’ve never seen a language where it feels completely elegant.

\n\n

We’ve got one more edge case to deal with before we end this chapter. Recall this strange beastie we first met in jlox’s implementation of variable resolution:

\n
{\n  var a = "outer";\n  {\n    var a = a;\n  }\n}\n
\n

We slayed it then by splitting a variable’s declaration into two phases, and\nwe’ll do that again here:

\"An\n

As soon as the variable declaration begins—in other words, before its\ninitializer—the name is declared in the current scope. The variable exists,\nbut in a special “uninitialized” state. Then we compile the initializer. If at\nany point in that expression we resolve an identifier that points back to this\nvariable, we’ll see that it is not initialized yet and report an error. After we\nfinish compiling the initializer, we mark the variable as initialized and ready\nfor use.

\n

To implement this, when we declare a local, we need to indicate the\n“uninitialized” state somehow. We could add a new field to Local, but let’s be a\nlittle more parsimonious with memory. Instead, we’ll set the variable’s scope\ndepth to a special sentinel value, -1.

\n
  local->name = name;\n
compiler.c
\nin addLocal()
\nreplace 1 line
\n
  local->depth = -1;\n
}\n
\n
compiler.c, in addLocal(), replace 1 line
\n\n

Later, once the variable’s initializer has been compiled, we mark it\ninitialized.

\n
  if (current->scopeDepth > 0) {\n
compiler.c
\nin defineVariable()
\n
    markInitialized();\n
    return;\n  }\n
\n
compiler.c, in defineVariable()
\n\n

That is implemented like so:

\n
compiler.c
\nadd after parseVariable()
\n
static void markInitialized() {\n  current->locals[current->localCount - 1].depth =\n      current->scopeDepth;\n}\n
\n
compiler.c, add after parseVariable()
\n\n

So this is really what “declaring” and “defining” a variable means in the\ncompiler. “Declaring” is when the variable is added to the scope, and “defining”\nis when it becomes available for use.

\n

When we resolve a reference to a local variable, we check the scope depth to see\nif it’s fully defined.

\n
    if (identifiersEqual(name, &local->name)) {\n
compiler.c
\nin resolveLocal()
\n
      if (local->depth == -1) {\n        error("Can't read local variable in its own initializer.");\n      }\n
      return i;\n
\n
compiler.c, in resolveLocal()
\n\n

If the variable has the sentinel depth, it must be a reference to a variable in\nits own initializer, and we report that as an error.

\n

That’s it for this chapter! We added blocks, local variables, and real,\nhonest-to-God lexical scoping. Given that we introduced an entirely different\nruntime representation for variables, we didn’t have to write a lot of code. The\nimplementation ended up being pretty clean and efficient.

\n

You’ll notice that almost all of the code we wrote is in the compiler. Over in\nthe runtime, it’s just two little instructions. You’ll see this as a continuing\ntrend in clox compared to jlox. One of the biggest\nhammers in the optimizer’s toolbox is pulling work forward into the compiler so\nthat you don’t have to do it at runtime. In this chapter, that meant resolving\nexactly which stack slot every local variable occupies. That way, at runtime, no\nlookup or resolution needs to happen.

\n\n
\n

Challenges

\n
    \n
  1. \n

    Our simple local array makes it easy to calculate the stack slot of each\nlocal variable. But it means that when the compiler resolves a reference to\na variable, we have to do a linear scan through the array.

    \n

    Come up with something more efficient. Do you think the additional\ncomplexity is worth it?

    \n
    2. \n
  2. \n

    How do other languages handle code like this:

    \n
    var a = a;\n
    \n

    What would you do if it was your language? Why?

    \n
    3. \n
  3. \n

    Many languages make a distinction between variables that can be reassigned\nand those that can’t. In Java, the final modifier prevents you from\nassigning to a variable. In JavaScript, a variable declared with let can\nbe assigned, but one declared using const can’t. Swift treats let as\nsingle-assignment and uses var for assignable variables. Scala and Kotlin\nuse val and var.

    \n

    Pick a keyword for a single-assignment variable form to add to Lox. Justify\nyour choice, then implement it. An attempt to assign to a variable declared\nusing your new keyword should cause a compile error.

    \n
    4. \n
  4. \n

    Extend clox to allow more than 256 local variables to be in scope at a time.

    \n
    5. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/methods-and-initializers.html", "content": "\n\n\n\nMethods and Initializers · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
28
\n

Methods and Initializers

\n\n
\n

When you are on the dancefloor, there is nothing to do but dance.

\n

Umberto Eco, The Mysterious Flame of Queen Loana

\n
\n

It is time for our virtual machine to bring its nascent objects to life with\nbehavior. That means methods and method calls. And, since they are a special\nkind of method, initializers too.

\n

All of this is familiar territory from our previous jlox interpreter. What’s new\nin this second trip is an important optimization we’ll implement to make method\ncalls over seven times faster than our baseline performance. But before we get\nto that fun, we gotta get the basic stuff working.

\n

28 . 1Method Declarations

\n

We can’t optimize method calls before we have method calls, and we can’t call\nmethods without having methods to call, so we’ll start with declarations.

\n

28 . 1 . 1Representing methods

\n

We usually start in the compiler, but let’s knock the object model out first\nthis time. The runtime representation for methods in clox is similar to that of\njlox. Each class stores a hash table of methods. Keys are method names, and each\nvalue is an ObjClosure for the body of the method.

\n
typedef struct {\n  Obj obj;\n  ObjString* name;\n
object.h
\nin struct ObjClass
\n
  Table methods;\n
} ObjClass;\n
\n
object.h, in struct ObjClass
\n\n

A brand new class begins with an empty method table.

\n
  klass->name = name; \n
object.c
\nin newClass()
\n
  initTable(&klass->methods);\n
  return klass;\n
\n
object.c, in newClass()
\n\n

The ObjClass struct owns the memory for this table, so when the memory manager\ndeallocates a class, the table should be freed too.

\n
    case OBJ_CLASS: {\n
memory.c
\nin freeObject()
\n
      ObjClass* klass = (ObjClass*)object;\n      freeTable(&klass->methods);\n
      FREE(ObjClass, object);\n
\n
memory.c, in freeObject()
\n\n

Speaking of memory managers, the GC needs to trace through classes into the\nmethod table. If a class is still reachable (likely through some instance),\nthen all of its methods certainly need to stick around too.

\n
      markObject((Obj*)klass->name);\n
memory.c
\nin blackenObject()
\n
      markTable(&klass->methods);\n
      break;\n
\n
memory.c, in blackenObject()
\n\n

We use the existing markTable() function, which traces through the key string\nand value in each table entry.

\n

Storing a class’s methods is pretty familiar coming from jlox. The different\npart is how that table gets populated. Our previous interpreter had access to\nthe entire AST node for the class declaration and all of the methods it\ncontained. At runtime, the interpreter simply walked that list of declarations.

\n

Now every piece of information the compiler wants to shunt over to the runtime\nhas to squeeze through the interface of a flat series of bytecode instructions.\nHow do we take a class declaration, which can contain an arbitrarily large set\nof methods, and represent it as bytecode? Let’s hop over to the compiler and\nfind out.

\n

28 . 1 . 2Compiling method declarations

\n

The last chapter left us with a compiler that parses classes but allows only an\nempty body. Now we insert a little code to compile a series of method\ndeclarations between the braces.

\n
  consume(TOKEN_LEFT_BRACE, "Expect '{' before class body.");\n
compiler.c
\nin classDeclaration()
\n
  while (!check(TOKEN_RIGHT_BRACE) && !check(TOKEN_EOF)) {\n    method();\n  }\n
  consume(TOKEN_RIGHT_BRACE, "Expect '}' after class body.");\n
\n
compiler.c, in classDeclaration()
\n\n

Lox doesn’t have field declarations, so anything before the closing brace at the\nend of the class body must be a method. We stop compiling methods when we hit\nthat final curly or if we reach the end of the file. The latter check ensures\nour compiler doesn’t get stuck in an infinite loop if the user accidentally\nforgets the closing brace.

\n

The tricky part with compiling a class declaration is that a class may declare\nany number of methods. Somehow the runtime needs to look up and bind all of\nthem. That would be a lot to pack into a single OP_CLASS instruction. Instead,\nthe bytecode we generate for a class declaration will split the process into a\nseries of instructions. The compiler already emits\nan OP_CLASS instruction that creates a new empty ObjClass object. Then it\nemits instructions to store the class in a variable with its name.

\n\n

Now, for each method declaration, we emit a new OP_METHOD instruction that\nadds a single method to that class. When all of the OP_METHOD instructions\nhave executed, we’re left with a fully formed class. While the user sees a class\ndeclaration as a single atomic operation, the VM implements it as a series of\nmutations.

\n

To define a new method, the VM needs three things:

\n
    \n
  1. \n

    The name of the method.

    \n
    2. \n
  2. \n

    The closure for the method body.

    \n
    3. \n
  3. \n

    The class to bind the method to.

    \n
    4. \n
\n

We’ll incrementally write the compiler code to see how those all get through to\nthe runtime, starting here:

\n
compiler.c
\nadd after function()
\n
static void method() {\n  consume(TOKEN_IDENTIFIER, "Expect method name.");\n  uint8_t constant = identifierConstant(&parser.previous);\n  emitBytes(OP_METHOD, constant);\n}\n
\n
compiler.c, add after function()
\n\n

Like OP_GET_PROPERTY and other instructions that need names at runtime, the\ncompiler adds the method name token’s lexeme to the constant table, getting back\na table index. Then we emit an OP_METHOD instruction with that index as the\noperand. That’s the name. Next is the method body:

\n
  uint8_t constant = identifierConstant(&parser.previous);\n
compiler.c
\nin method()
\n
\n\n  FunctionType type = TYPE_FUNCTION;\n  function(type);\n
  emitBytes(OP_METHOD, constant);\n
\n
compiler.c, in method()
\n\n

We use the same function() helper that we wrote for compiling function\ndeclarations. That utility function compiles the subsequent parameter list and\nfunction body. Then it emits the code to create an ObjClosure and leave it on\ntop of the stack. At runtime, the VM will find the closure there.

\n

Last is the class to bind the method to. Where can the VM find that?\nUnfortunately, by the time we reach the OP_METHOD instruction, we don’t know\nwhere it is. It could be on the stack, if the user\ndeclared the class in a local scope. But a top-level class declaration ends up\nwith the ObjClass in the global variable table.

\n\n

Fear not. The compiler does know the name of the class. We can capture it\nright after we consume its token.

\n
  consume(TOKEN_IDENTIFIER, "Expect class name.");\n
compiler.c
\nin classDeclaration()
\n
  Token className = parser.previous;\n
  uint8_t nameConstant = identifierConstant(&parser.previous);\n
\n
compiler.c, in classDeclaration()
\n\n

And we know that no other declaration with that name could possibly shadow the\nclass. So we do the easy fix. Before we start binding methods, we emit whatever\ncode is necessary to load the class back on top of the stack.

\n
  defineVariable(nameConstant);\n\n
compiler.c
\nin classDeclaration()
\n
  namedVariable(className, false);\n
  consume(TOKEN_LEFT_BRACE, "Expect '{' before class body.");\n
\n
compiler.c, in classDeclaration()
\n\n

Right before compiling the class body, we call\nnamedVariable(). That helper function generates code to load a variable with\nthe given name onto the stack. Then we compile the methods.

\n\n

This means that when we execute each OP_METHOD instruction, the stack has the\nmethod’s closure on top with the class right under it. Once we’ve reached the\nend of the methods, we no longer need the class and tell the VM to pop it off\nthe stack.

\n
  consume(TOKEN_RIGHT_BRACE, "Expect '}' after class body.");\n
compiler.c
\nin classDeclaration()
\n
  emitByte(OP_POP);\n
}\n
\n
compiler.c, in classDeclaration()
\n\n

Putting all of that together, here is an example class declaration to throw at\nthe compiler:

\n
class Brunch {\n  bacon() {}\n  eggs() {}\n}\n
\n

Given that, here is what the compiler generates and how those instructions\naffect the stack at runtime:

\"The\n

All that remains for us is to implement the runtime for that new OP_METHOD\ninstruction.

\n

28 . 1 . 3Executing method declarations

\n

First we define the opcode.

\n
  OP_CLASS,\n
chunk.h
\nin enum OpCode
\n
  OP_METHOD\n
} OpCode;\n
\n
chunk.h, in enum OpCode
\n\n

We disassemble it like other instructions that have string constant operands.

\n
    case OP_CLASS:\n      return constantInstruction("OP_CLASS", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_METHOD:\n      return constantInstruction("OP_METHOD", chunk, offset);\n
    default:\n
\n
debug.c, in disassembleInstruction()
\n\n

And over in the interpreter, we add a new case too.

\n
        break;\n
vm.c
\nin run()
\n
      case OP_METHOD:\n        defineMethod(READ_STRING());\n        break;\n
    }\n
\n
vm.c, in run()
\n\n

There, we read the method name from the constant table and pass it here:

\n
vm.c
\nadd after closeUpvalues()
\n
static void defineMethod(ObjString* name) {\n  Value method = peek(0);\n  ObjClass* klass = AS_CLASS(peek(1));\n  tableSet(&klass->methods, name, method);\n  pop();\n}\n
\n
vm.c, add after closeUpvalues()
\n\n

The method closure is on top of the stack, above the class it will be bound to.\nWe read those two stack slots and store the closure in the class’s method table.\nThen we pop the closure since we’re done with it.

\n

Note that we don’t do any runtime type checking on the closure or class object.\nThat AS_CLASS() call is safe because the compiler itself generated the code\nthat causes the class to be in that stack slot. The VM trusts its own compiler.

\n\n

After the series of OP_METHOD instructions is done and the OP_POP has popped\nthe class, we will have a class with a nicely populated method table, ready to\nstart doing things. The next step is pulling those methods back out and using\nthem.

\n

28 . 2Method References

\n

Most of the time, methods are accessed and immediately called, leading to this\nfamiliar syntax:

\n
instance.method(argument);\n
\n

But remember, in Lox and some other languages, those two steps are distinct and\ncan be separated.

\n
var closure = instance.method;\nclosure(argument);\n
\n

Since users can separate the operations, we have to implement them separately.\nThe first step is using our existing dotted property syntax to access a method\ndefined on the instance’s class. That should return some kind of object that the\nuser can then call like a function.

\n

The obvious approach is to look up the method in the class’s method table and\nreturn the ObjClosure associated with that name. But we also need to remember\nthat when you access a method, this gets bound to the instance the method was\naccessed from. Here’s the example from when we added methods to jlox:

\n
class Person {\n  sayName() {\n    print this.name;\n  }\n}\n\nvar jane = Person();\njane.name = "Jane";\n\nvar method = jane.sayName;\nmethod(); // ?\n
\n

This should print “Jane”, so the object returned by .sayName somehow needs to\nremember the instance it was accessed from when it later gets called. In jlox,\nwe implemented that “memory” using the interpreter’s existing heap-allocated\nEnvironment class, which handled all variable storage.

\n

Our bytecode VM has a more complex architecture for storing state. Local\nvariables and temporaries are on the stack, globals are in a hash\ntable, and variables in closures use upvalues. That necessitates a somewhat\nmore complex solution for tracking a method’s receiver in clox, and a new\nruntime type.

\n

28 . 2 . 1Bound methods

\n

When the user executes a method access, we’ll find the closure for that method\nand wrap it in a new “bound method” object that tracks\nthe instance that the method was accessed from. This bound object can be called\nlater like a function. When invoked, the VM will do some shenanigans to wire up\nthis to point to the receiver inside the method’s body.

\n\n

Here’s the new object type:

\n
} ObjInstance;\n\n
object.h
\nadd after struct ObjInstance
\n
typedef struct {\n  Obj obj;\n  Value receiver;\n  ObjClosure* method;\n} ObjBoundMethod;\n\n
ObjClass* newClass(ObjString* name);\n
\n
object.h, add after struct ObjInstance
\n\n

It wraps the receiver and the method closure together. The receiver’s type is\nValue even though methods can be called only on ObjInstances. Since the VM\ndoesn’t care what kind of receiver it has anyway, using Value means we don’t\nhave to keep converting the pointer back to a Value when it gets passed to more\ngeneral functions.

\n

The new struct implies the usual boilerplate you’re used to by now. A new case\nin the object type enum:

\n
typedef enum {\n
object.h
\nin enum ObjType
\n
  OBJ_BOUND_METHOD,\n
  OBJ_CLASS,\n
\n
object.h, in enum ObjType
\n\n

A macro to check a value’s type:

\n
#define OBJ_TYPE(value)        (AS_OBJ(value)->type)\n\n
object.h
\n
#define IS_BOUND_METHOD(value) isObjType(value, OBJ_BOUND_METHOD)\n
#define IS_CLASS(value)        isObjType(value, OBJ_CLASS)\n
\n
object.h
\n\n

Another macro to cast the value to an ObjBoundMethod pointer:

\n
#define IS_STRING(value)       isObjType(value, OBJ_STRING)\n\n
object.h
\n
#define AS_BOUND_METHOD(value) ((ObjBoundMethod*)AS_OBJ(value))\n
#define AS_CLASS(value)        ((ObjClass*)AS_OBJ(value))\n
\n
object.h
\n\n

A function to create a new ObjBoundMethod:

\n
} ObjBoundMethod;\n\n
object.h
\nadd after struct ObjBoundMethod
\n
ObjBoundMethod* newBoundMethod(Value receiver,\n                               ObjClosure* method);\n
ObjClass* newClass(ObjString* name);\n
\n
object.h, add after struct ObjBoundMethod
\n\n

And an implementation of that function here:

\n
object.c
\nadd after allocateObject()
\n
ObjBoundMethod* newBoundMethod(Value receiver,\n                               ObjClosure* method) {\n  ObjBoundMethod* bound = ALLOCATE_OBJ(ObjBoundMethod,\n                                       OBJ_BOUND_METHOD);\n  bound->receiver = receiver;\n  bound->method = method;\n  return bound;\n}\n
\n
object.c, add after allocateObject()
\n\n

The constructor-like function simply stores the given closure and receiver. When\nthe bound method is no longer needed, we free it.

\n
  switch (object->type) {\n
memory.c
\nin freeObject()
\n
    case OBJ_BOUND_METHOD:\n      FREE(ObjBoundMethod, object);\n      break;\n
    case OBJ_CLASS: {\n
\n
memory.c, in freeObject()
\n\n

The bound method has a couple of references, but it doesn’t own them, so it\nfrees nothing but itself. However, those references do get traced by the garbage\ncollector.

\n
  switch (object->type) {\n
memory.c
\nin blackenObject()
\n
    case OBJ_BOUND_METHOD: {\n      ObjBoundMethod* bound = (ObjBoundMethod*)object;\n      markValue(bound->receiver);\n      markObject((Obj*)bound->method);\n      break;\n    }\n
    case OBJ_CLASS: {\n
\n
memory.c, in blackenObject()
\n\n

This ensures that a handle to a method keeps the\nreceiver around in memory so that this can still find the object when you\ninvoke the handle later. We also trace the method closure.

\n\n

The last operation all objects support is printing.

\n
  switch (OBJ_TYPE(value)) {\n
object.c
\nin printObject()
\n
    case OBJ_BOUND_METHOD:\n      printFunction(AS_BOUND_METHOD(value)->method->function);\n      break;\n
    case OBJ_CLASS:\n
\n
object.c, in printObject()
\n\n

A bound method prints exactly the same way as a function. From the user’s\nperspective, a bound method is a function. It’s an object they can call. We\ndon’t expose that the VM implements bound methods using a different object type.

\n\n

Put on your party hat because we just reached a little\nmilestone. ObjBoundMethod is the very last runtime type to add to clox. You’ve\nwritten your last IS_ and AS_ macros. We’re only a few chapters from the end\nof the book, and we’re getting close to a complete VM.

\n

28 . 2 . 2Accessing methods

\n

Let’s get our new object type doing something. Methods are accessed using the\nsame “dot” property syntax we implemented in the last chapter. The compiler\nalready parses the right expressions and emits OP_GET_PROPERTY instructions\nfor them. The only changes we need to make are in the runtime.

\n

When a property access instruction executes, the instance is on top of the\nstack. The instruction’s job is to find a field or method with the given name\nand replace the top of the stack with the accessed property.

\n

The interpreter already handles fields, so we simply extend the\nOP_GET_PROPERTY case with another section.

\n
          pop(); // Instance.\n          push(value);\n          break;\n        }\n\n
vm.c
\nin run()
\nreplace 2 lines
\n
        if (!bindMethod(instance->klass, name)) {\n          return INTERPRET_RUNTIME_ERROR;\n        }\n        break;\n
      }\n
\n
vm.c, in run(), replace 2 lines
\n\n

We insert this after the code to look up a field on the receiver instance.\nFields take priority over and shadow methods, so we look for a field first. If\nthe instance does not have a field with the given property name, then the name\nmay refer to a method.

\n

We take the instance’s class and pass it to a new bindMethod() helper. If that\nfunction finds a method, it places the method on the stack and returns true.\nOtherwise it returns false to indicate a method with that name couldn’t be\nfound. Since the name also wasn’t a field, that means we have a runtime error,\nwhich aborts the interpreter.

\n

Here is the good stuff:

\n
vm.c
\nadd after callValue()
\n
static bool bindMethod(ObjClass* klass, ObjString* name) {\n  Value method;\n  if (!tableGet(&klass->methods, name, &method)) {\n    runtimeError("Undefined property '%s'.", name->chars);\n    return false;\n  }\n\n  ObjBoundMethod* bound = newBoundMethod(peek(0),\n                                         AS_CLOSURE(method));\n  pop();\n  push(OBJ_VAL(bound));\n  return true;\n}\n
\n
vm.c, add after callValue()
\n\n

First we look for a method with the given name in the class’s method table. If\nwe don’t find one, we report a runtime error and bail out. Otherwise, we take\nthe method and wrap it in a new ObjBoundMethod. We grab the receiver from its\nhome on top of the stack. Finally, we pop the instance and replace the top of\nthe stack with the bound method.

\n

For example:

\n
class Brunch {\n  eggs() {}\n}\n\nvar brunch = Brunch();\nvar eggs = brunch.eggs;\n
\n

Here is what happens when the VM executes the bindMethod() call for the\nbrunch.eggs expression:

\"The\n

That’s a lot of machinery under the hood, but from the user’s perspective, they\nsimply get a function that they can call.

\n

28 . 2 . 3Calling methods

\n

Users can declare methods on classes, access them on instances, and get bound\nmethods onto the stack. They just can’t do anything\nuseful with those bound method objects. The operation we’re missing is calling\nthem. Calls are implemented in callValue(), so we add a case there for the new\nobject type.

\n\n
    switch (OBJ_TYPE(callee)) {\n
vm.c
\nin callValue()
\n
      case OBJ_BOUND_METHOD: {\n        ObjBoundMethod* bound = AS_BOUND_METHOD(callee);\n        return call(bound->method, argCount);\n      }\n
      case OBJ_CLASS: {\n
\n
vm.c, in callValue()
\n\n

We pull the raw closure back out of the ObjBoundMethod and use the existing\ncall() helper to begin an invocation of that closure by pushing a CallFrame\nfor it onto the call stack. That’s all it takes to be able to run this Lox\nprogram:

\n
class Scone {\n  topping(first, second) {\n    print "scone with " + first + " and " + second;\n  }\n}\n\nvar scone = Scone();\nscone.topping("berries", "cream");\n
\n

That’s three big steps. We can declare, access, and invoke methods. But\nsomething is missing. We went to all that trouble to wrap the method closure in\nan object that binds the receiver, but when we invoke the method, we don’t use\nthat receiver at all.

\n

28 . 3This

\n

The reason bound methods need to keep hold of the receiver is so that it can be\naccessed inside the body of the method. Lox exposes a method’s receiver through\nthis expressions. It’s time for some new syntax. The lexer already treats\nthis as a special token type, so the first step is wiring that token up in the\nparse table.

\n
  [TOKEN_SUPER]         = {NULL,     NULL,   PREC_NONE},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_THIS]          = {this_,    NULL,   PREC_NONE},\n
  [TOKEN_TRUE]          = {literal,  NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n\n

When the parser encounters a this in prefix position, it dispatches to a new\nparser function.

\n
compiler.c
\nadd after variable()
\n
static void this_(bool canAssign) {\n  variable(false);\n} \n
\n
compiler.c, add after variable()
\n\n

We’ll apply the same implementation technique for this in clox that we used in\njlox. We treat this as a lexically scoped local variable whose value gets\nmagically initialized. Compiling it like a local variable means we get a lot of\nbehavior for free. In particular, closures inside a method that reference this\nwill do the right thing and capture the receiver in an upvalue.

\n

When the parser function is called, the this token has just been consumed and\nis stored as the previous token. We call our existing variable() function\nwhich compiles identifier expressions as variable accesses. It takes a single\nBoolean parameter for whether the compiler should look for a following =\noperator and parse a setter. You can’t assign to this, so we pass false to\ndisallow that.

\n

The variable() function doesn’t care that this has its own token type and\nisn’t an identifier. It is happy to treat the lexeme “this” as if it were a\nvariable name and then look it up using the existing scope resolution machinery.\nRight now, that lookup will fail because we never declared a variable whose name\nis “this”. It’s time to think about where the receiver should live in memory.

\n

At least until they get captured by closures, clox stores every local variable\non the VM’s stack. The compiler keeps track of which slots in the function’s\nstack window are owned by which local variables. If you recall, the compiler\nsets aside stack slot zero by declaring a local variable whose name is an empty\nstring.

\n

For function calls, that slot ends up holding the function being called. Since\nthe slot has no name, the function body never accesses it. You can guess where\nthis is going. For method calls, we can repurpose that slot to store the\nreceiver. Slot zero will store the instance that this is bound to. In order to\ncompile this expressions, the compiler simply needs to give the correct name\nto that local variable.

\n
  local->isCaptured = false;\n
compiler.c
\nin initCompiler()
\nreplace 2 lines
\n
  if (type != TYPE_FUNCTION) {\n    local->name.start = "this";\n    local->name.length = 4;\n  } else {\n    local->name.start = "";\n    local->name.length = 0;\n  }\n
}\n
\n
compiler.c, in initCompiler(), replace 2 lines
\n\n

We want to do this only for methods. Function declarations don’t have a this.\nAnd, in fact, they must not declare a variable named “this”, so that if you\nwrite a this expression inside a function declaration which is itself inside a\nmethod, the this correctly resolves to the outer method’s receiver.

\n
class Nested {\n  method() {\n    fun function() {\n      print this;\n    }\n\n    function();\n  }\n}\n\nNested().method();\n
\n

This program should print “Nested instance”. To decide what name to give to\nlocal slot zero, the compiler needs to know whether it’s compiling a function or\nmethod declaration, so we add a new case to our FunctionType enum to distinguish\nmethods.

\n
  TYPE_FUNCTION,\n
compiler.c
\nin enum FunctionType
\n
  TYPE_METHOD,\n
  TYPE_SCRIPT\n
\n
compiler.c, in enum FunctionType
\n\n

When we compile a method, we use that type.

\n
  uint8_t constant = identifierConstant(&parser.previous);\n\n
compiler.c
\nin method()
\nreplace 1 line
\n
  FunctionType type = TYPE_METHOD;\n
  function(type);\n
\n
compiler.c, in method(), replace 1 line
\n\n

Now we can correctly compile references to the special “this” variable, and the\ncompiler will emit the right OP_GET_LOCAL instructions to access it. Closures\ncan even capture this and store the receiver in upvalues. Pretty cool.

\n

Except that at runtime, the receiver isn’t actually in slot zero. The\ninterpreter isn’t holding up its end of the bargain yet. Here is the fix:

\n
      case OBJ_BOUND_METHOD: {\n        ObjBoundMethod* bound = AS_BOUND_METHOD(callee);\n
vm.c
\nin callValue()
\n
        vm.stackTop[-argCount - 1] = bound->receiver;\n
        return call(bound->method, argCount);\n      }\n
\n
vm.c, in callValue()
\n\n

When a method is called, the top of the stack contains all of the arguments, and\nthen just under those is the closure of the called method. That’s where slot\nzero in the new CallFrame will be. This line of code inserts the receiver into\nthat slot. For example, given a method call like this:

\n
scone.topping("berries", "cream");\n
\n

We calculate the slot to store the receiver like so:

\"Skipping\n

The -argCount skips past the arguments and the - 1 adjusts for the fact that\nstackTop points just past the last used stack slot.

\n

28 . 3 . 1Misusing this

\n

Our VM now supports users correctly using this, but we also need to make\nsure it properly handles users misusing this. Lox says it is a compile\nerror for a this expression to appear outside of the body of a method. These\ntwo wrong uses should be caught by the compiler:

\n
print this; // At top level.\n\nfun notMethod() {\n  print this; // In a function.\n}\n
\n

So how does the compiler know if it’s inside a method? The obvious answer is to\nlook at the FunctionType of the current Compiler. We did just add an enum case\nthere to treat methods specially. However, that wouldn’t correctly handle code\nlike the earlier example where you are inside a function which is, itself,\nnested inside a method.

\n

We could try to resolve “this” and then report an error if it wasn’t found in\nany of the surrounding lexical scopes. That would work, but would require us to\nshuffle around a bunch of code, since right now the code for resolving a\nvariable implicitly considers it a global access if no declaration is found.

\n

In the next chapter, we will need information about the nearest enclosing class.\nIf we had that, we could use it here to determine if we are inside a method. So\nwe may as well make our future selves’ lives a little easier and put that\nmachinery in place now.

\n
Compiler* current = NULL;\n
compiler.c
\nadd after variable current
\n
ClassCompiler* currentClass = NULL;\n
\n\nstatic Chunk* currentChunk() {\n
\n
compiler.c, add after variable current
\n\n

This module variable points to a struct representing the current, innermost\nclass being compiled. The new type looks like this:

\n
} Compiler;\n
compiler.c
\nadd after struct Compiler
\n
\n\ntypedef struct ClassCompiler {\n  struct ClassCompiler* enclosing;\n} ClassCompiler;\n
\n\nParser parser;\n
\n
compiler.c, add after struct Compiler
\n\n

Right now we store only a pointer to the ClassCompiler for the enclosing class,\nif any. Nesting a class declaration inside a method in some other class is an\nuncommon thing to do, but Lox supports it. Just like the Compiler struct, this\nmeans ClassCompiler forms a linked list from the current innermost class being\ncompiled out through all of the enclosing classes.

\n

If we aren’t inside any class declaration at all, the module variable\ncurrentClass is NULL. When the compiler begins compiling a class, it pushes\na new ClassCompiler onto that implicit linked stack.

\n
  defineVariable(nameConstant);\n\n
compiler.c
\nin classDeclaration()
\n
  ClassCompiler classCompiler;\n  classCompiler.enclosing = currentClass;\n  currentClass = &classCompiler;\n\n
  namedVariable(className, false);\n
\n
compiler.c, in classDeclaration()
\n\n

The memory for the ClassCompiler struct lives right on the C stack, a handy\ncapability we get by writing our compiler using recursive descent. At the end of\nthe class body, we pop that compiler off the stack and restore the enclosing\none.

\n
  emitByte(OP_POP);\n
compiler.c
\nin classDeclaration()
\n
\n\n  currentClass = currentClass->enclosing;\n
}\n
\n
compiler.c, in classDeclaration()
\n\n

When an outermost class body ends, enclosing will be NULL, so this resets\ncurrentClass to NULL. Thus, to see if we are inside a class—and therefore\ninside a method—we simply check that module variable.

\n
static void this_(bool canAssign) {\n
compiler.c
\nin this_()
\n
  if (currentClass == NULL) {\n    error("Can't use 'this' outside of a class.");\n    return;\n  }\n\n
  variable(false);\n
\n
compiler.c, in this_()
\n\n

With that, this outside of a class is correctly forbidden. Now our methods\nreally feel like methods in the object-oriented sense. Accessing the receiver\nlets them affect the instance you called the method on. We’re getting there!

\n

28 . 4Instance Initializers

\n

The reason object-oriented languages tie state and behavior together—one of\nthe core tenets of the paradigm—is to ensure that objects are always in a\nvalid, meaningful state. When the only way to touch an object’s state is through its methods, the methods can make sure nothing\ngoes awry. But that presumes the object is already in a proper state. What\nabout when it’s first created?

\n\n

Object-oriented languages ensure that brand new objects are properly set up\nthrough constructors, which both produce a new instance and initialize its\nstate. In Lox, the runtime allocates new raw instances, and a class may declare\nan initializer to set up any fields. Initializers work mostly like normal\nmethods, with a few tweaks:

\n
    \n
  1. \n

    The runtime automatically invokes the initializer method whenever an\ninstance of a class is created.

    \n
    2. \n
  2. \n

    The caller that constructs an instance always gets the instance back after the initializer finishes, regardless of what\nthe initializer function itself returns. The initializer method doesn’t need\nto explicitly return this.

    \n
    3. \n
  3. \n

    In fact, an initializer is prohibited from returning any value at all\nsince the value would never be seen anyway.

    \n
    4. \n
\n\n

Now that we support methods, to add initializers, we merely need to implement\nthose three special rules. We’ll go in order.

\n

28 . 4 . 1Invoking initializers

\n

First, automatically calling init() on new instances:

\n
        vm.stackTop[-argCount - 1] = OBJ_VAL(newInstance(klass));\n
vm.c
\nin callValue()
\n
        Value initializer;\n        if (tableGet(&klass->methods, vm.initString,\n                     &initializer)) {\n          return call(AS_CLOSURE(initializer), argCount);\n        }\n
        return true;\n
\n
vm.c, in callValue()
\n\n

After the runtime allocates the new instance, we look for an init() method on\nthe class. If we find one, we initiate a call to it. This pushes a new CallFrame\nfor the initializer’s closure. Say we run this program:

\n
class Brunch {\n  init(food, drink) {}\n}\n\nBrunch("eggs", "coffee");\n
\n

When the VM executes the call to Brunch(), it goes like this:

\"The\n

Any arguments passed to the class when we called it are still sitting on the\nstack above the instance. The new CallFrame for the init() method shares that\nstack window, so those arguments implicitly get forwarded to the initializer.

\n

Lox doesn’t require a class to define an initializer. If omitted, the runtime\nsimply returns the new uninitialized instance. However, if there is no init()\nmethod, then it doesn’t make any sense to pass arguments to the class when\ncreating the instance. We make that an error.

\n
          return call(AS_CLOSURE(initializer), argCount);\n
vm.c
\nin callValue()
\n
        } else if (argCount != 0) {\n          runtimeError("Expected 0 arguments but got %d.",\n                       argCount);\n          return false;\n
        }\n
\n
vm.c, in callValue()
\n\n

When the class does provide an initializer, we also need to ensure that the\nnumber of arguments passed matches the initializer’s arity. Fortunately, the\ncall() helper does that for us already.

\n

To call the initializer, the runtime looks up the init() method by name. We\nwant that to be fast since it happens every time an instance is constructed.\nThat means it would be good to take advantage of the string interning we’ve\nalready implemented. To do that, the VM creates an ObjString for “init” and\nreuses it. The string lives right in the VM struct.

\n
  Table strings;\n
vm.h
\nin struct VM
\n
  ObjString* initString;\n
  ObjUpvalue* openUpvalues;\n
\n
vm.h, in struct VM
\n\n

We create and intern the string when the VM boots up.

\n
  initTable(&vm.strings);\n
vm.c
\nin initVM()
\n
\n\n  vm.initString = copyString("init", 4);\n
\n\n  defineNative("clock", clockNative);\n
\n
vm.c, in initVM()
\n\n

We want it to stick around, so the GC considers it a root.

\n
  markCompilerRoots();\n
memory.c
\nin markRoots()
\n
  markObject((Obj*)vm.initString);\n
}\n
\n
memory.c, in markRoots()
\n\n

Look carefully. See any bug waiting to happen? No? It’s a subtle one. The\ngarbage collector now reads vm.initString. That field is initialized from the\nresult of calling copyString(). But copying a string allocates memory, which\ncan trigger a GC. If the collector ran at just the wrong time, it would read\nvm.initString before it had been initialized. So, first we zero the field out.

\n
  initTable(&vm.strings);\n\n
vm.c
\nin initVM()
\n
  vm.initString = NULL;\n
  vm.initString = copyString("init", 4);\n\n
\n
vm.c, in initVM()
\n\n

We clear the pointer when the VM shuts down since the next line will free it.

\n
  freeTable(&vm.strings);\n
vm.c
\nin freeVM()
\n
  vm.initString = NULL;\n
  freeObjects();\n
\n
vm.c, in freeVM()
\n\n

OK, that lets us call initializers.

\n

28 . 4 . 2Initializer return values

\n

The next step is ensuring that constructing an instance of a class with an\ninitializer always returns the new instance, and not nil or whatever the body\nof the initializer returns. Right now, if a class defines an initializer, then\nwhen an instance is constructed, the VM pushes a call to that initializer onto\nthe CallFrame stack. Then it just keeps on trucking.

\n

The user’s invocation on the class to create the instance will complete whenever\nthat initializer method returns, and will leave on the stack whatever value the\ninitializer puts there. That means that unless the user takes care to put\nreturn this; at the end of the initializer, no instance will come out. Not\nvery helpful.

\n

To fix this, whenever the front end compiles an initializer method, it will emit\ndifferent bytecode at the end of the body to return this from the method\ninstead of the usual implicit nil most functions return. In order to do\nthat, the compiler needs to actually know when it is compiling an initializer.\nWe detect that by checking to see if the name of the method we’re compiling is\n“init”.

\n
  FunctionType type = TYPE_METHOD;\n
compiler.c
\nin method()
\n
  if (parser.previous.length == 4 &&\n      memcmp(parser.previous.start, "init", 4) == 0) {\n    type = TYPE_INITIALIZER;\n  }\n\n
  function(type);\n
\n
compiler.c, in method()
\n\n

We define a new function type to distinguish initializers from other methods.

\n
  TYPE_FUNCTION,\n
compiler.c
\nin enum FunctionType
\n
  TYPE_INITIALIZER,\n
  TYPE_METHOD,\n
\n
compiler.c, in enum FunctionType
\n\n

Whenever the compiler emits the implicit return at the end of a body, we check\nthe type to decide whether to insert the initializer-specific behavior.

\n
static void emitReturn() {\n
compiler.c
\nin emitReturn()
\nreplace 1 line
\n
  if (current->type == TYPE_INITIALIZER) {\n    emitBytes(OP_GET_LOCAL, 0);\n  } else {\n    emitByte(OP_NIL);\n  }\n\n
  emitByte(OP_RETURN);\n
\n
compiler.c, in emitReturn(), replace 1 line
\n\n

In an initializer, instead of pushing nil onto the stack before returning,\nwe load slot zero, which contains the instance. This emitReturn() function is\nalso called when compiling a return statement without a value, so this also\ncorrectly handles cases where the user does an early return inside the\ninitializer.

\n

28 . 4 . 3Incorrect returns in initializers

\n

The last step, the last item in our list of special features of initializers, is\nmaking it an error to try to return anything else from an initializer. Now\nthat the compiler tracks the method type, this is straightforward.

\n
  if (match(TOKEN_SEMICOLON)) {\n    emitReturn();\n  } else {\n
compiler.c
\nin returnStatement()
\n
    if (current->type == TYPE_INITIALIZER) {\n      error("Can't return a value from an initializer.");\n    }\n\n
    expression();\n
\n
compiler.c, in returnStatement()
\n\n

We report an error if a return statement in an initializer has a value. We\nstill go ahead and compile the value afterwards so that the compiler doesn’t get\nconfused by the trailing expression and report a bunch of cascaded errors.

\n

Aside from inheritance, which we’ll get to soon, we now have a\nfairly full-featured class system working in clox.

\n
class CoffeeMaker {\n  init(coffee) {\n    this.coffee = coffee;\n  }\n\n  brew() {\n    print "Enjoy your cup of " + this.coffee;\n\n    // No reusing the grounds!\n    this.coffee = nil;\n  }\n}\n\nvar maker = CoffeeMaker("coffee and chicory");\nmaker.brew();\n
\n

Pretty fancy for a C program that would fit on an old floppy disk.

\n\n

28 . 5Optimized Invocations

\n

Our VM correctly implements the language’s semantics for method calls and\ninitializers. We could stop here. But the main reason we are building an entire\nsecond implementation of Lox from scratch is to execute faster than our old Java\ninterpreter. Right now, method calls even in clox are slow.

\n

Lox’s semantics define a method invocation as two operations—accessing the\nmethod and then calling the result. Our VM must support those as separate\noperations because the user can separate them. You can access a method without\ncalling it and then invoke the bound method later. Nothing we’ve implemented so\nfar is unnecessary.

\n

But always executing those as separate operations has a significant cost.\nEvery single time a Lox program accesses and invokes a method, the runtime\nheap allocates a new ObjBoundMethod, initializes its fields, then pulls them\nright back out. Later, the GC has to spend time freeing all of those ephemeral\nbound methods.

\n

Most of the time, a Lox program accesses a method and then immediately calls it.\nThe bound method is created by one bytecode instruction and then consumed by the\nvery next one. In fact, it’s so immediate that the compiler can even textually\nsee that it’s happening—a dotted property access followed by an opening\nparenthesis is most likely a method call.

\n

Since we can recognize this pair of operations at compile time, we have the\nopportunity to emit a new, special instruction that\nperforms an optimized method call.

\n

We start in the function that compiles dotted property expressions.

\n\n
  if (canAssign && match(TOKEN_EQUAL)) {\n    expression();\n    emitBytes(OP_SET_PROPERTY, name);\n
compiler.c
\nin dot()
\n
  } else if (match(TOKEN_LEFT_PAREN)) {\n    uint8_t argCount = argumentList();\n    emitBytes(OP_INVOKE, name);\n    emitByte(argCount);\n
  } else {\n
\n
compiler.c, in dot()
\n\n

After the compiler has parsed the property name, we look for a left parenthesis.\nIf we match one, we switch to a new code path. There, we compile the argument\nlist exactly like we do when compiling a call expression. Then we emit a single\nnew OP_INVOKE instruction. It takes two operands:

\n
    \n
  1. \n

    The index of the property name in the constant table.

    \n
    2. \n
  2. \n

    The number of arguments passed to the method.

    \n
    3. \n
\n

In other words, this single instruction combines the operands of the\nOP_GET_PROPERTY and OP_CALL instructions it replaces, in that order. It\nreally is a fusion of those two instructions. Let’s define it.

\n
  OP_CALL,\n
chunk.h
\nin enum OpCode
\n
  OP_INVOKE,\n
  OP_CLOSURE,\n
\n
chunk.h, in enum OpCode
\n\n

And add it to the disassembler:

\n
    case OP_CALL:\n      return byteInstruction("OP_CALL", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_INVOKE:\n      return invokeInstruction("OP_INVOKE", chunk, offset);\n
    case OP_CLOSURE: {\n
\n
debug.c, in disassembleInstruction()
\n\n

This is a new, special instruction format, so it needs a little custom\ndisassembly logic.

\n
debug.c
\nadd after constantInstruction()
\n
static int invokeInstruction(const char* name, Chunk* chunk,\n                                int offset) {\n  uint8_t constant = chunk->code[offset + 1];\n  uint8_t argCount = chunk->code[offset + 2];\n  printf("%-16s (%d args) %4d '", name, argCount, constant);\n  printValue(chunk->constants.values[constant]);\n  printf("'\\n");\n  return offset + 3;\n}\n
\n
debug.c, add after constantInstruction()
\n\n

We read the two operands and then print out both the method name and the\nargument count. Over in the interpreter’s bytecode dispatch loop is where the\nreal action begins.

\n
      }\n
vm.c
\nin run()
\n
      case OP_INVOKE: {\n        ObjString* method = READ_STRING();\n        int argCount = READ_BYTE();\n        if (!invoke(method, argCount)) {\n          return INTERPRET_RUNTIME_ERROR;\n        }\n        frame = &vm.frames[vm.frameCount - 1];\n        break;\n      }\n
      case OP_CLOSURE: {\n
\n
vm.c, in run()
\n\n

Most of the work happens in invoke(), which we’ll get to. Here, we look up the\nmethod name from the first operand and then read the argument count operand.\nThen we hand off to invoke() to do the heavy lifting. That function returns\ntrue if the invocation succeeds. As usual, a false return means a runtime\nerror occurred. We check for that here and abort the interpreter if disaster has\nstruck.

\n

Finally, assuming the invocation succeeded, then there is a new CallFrame on the\nstack, so we refresh our cached copy of the current frame in frame.

\n

The interesting work happens here:

\n
vm.c
\nadd after callValue()
\n
static bool invoke(ObjString* name, int argCount) {\n  Value receiver = peek(argCount);\n  ObjInstance* instance = AS_INSTANCE(receiver);\n  return invokeFromClass(instance->klass, name, argCount);\n}\n
\n
vm.c, add after callValue()
\n\n

First we grab the receiver off the stack. The arguments passed to the method are\nabove it on the stack, so we peek that many slots down. Then it’s a simple\nmatter to cast the object to an instance and invoke the method on it.

\n

That does assume the object is an instance. As with OP_GET_PROPERTY\ninstructions, we also need to handle the case where a user incorrectly tries to\ncall a method on a value of the wrong type.

\n
  Value receiver = peek(argCount);\n
vm.c
\nin invoke()
\n
\n\n  if (!IS_INSTANCE(receiver)) {\n    runtimeError("Only instances have methods.");\n    return false;\n  }\n\n
  ObjInstance* instance = AS_INSTANCE(receiver);\n
\n
vm.c, in invoke()
\n\n

That’s a runtime error, so we report that and bail\nout. Otherwise, we get the instance’s class and jump over to this other new\nutility function:

\n\n
vm.c
\nadd after callValue()
\n
static bool invokeFromClass(ObjClass* klass, ObjString* name,\n                            int argCount) {\n  Value method;\n  if (!tableGet(&klass->methods, name, &method)) {\n    runtimeError("Undefined property '%s'.", name->chars);\n    return false;\n  }\n  return call(AS_CLOSURE(method), argCount);\n}\n
\n
vm.c, add after callValue()
\n\n

This function combines the logic of how the VM implements OP_GET_PROPERTY and\nOP_CALL instructions, in that order. First we look up the method by name in\nthe class’s method table. If we don’t find one, we report that runtime error and\nexit.

\n

Otherwise, we take the method’s closure and push a call to it onto the CallFrame\nstack. We don’t need to heap allocate and initialize an ObjBoundMethod. In fact,\nwe don’t even need to juggle anything on the stack.\nThe receiver and method arguments are already right where they need to be.

\n\n

If you fire up the VM and run a little program that calls methods now, you\nshould see the exact same behavior as before. But, if we did our job right, the\nperformance should be much improved. I wrote a little microbenchmark that\ndoes a batch of 10,000 method calls. Then it tests how many of these batches it\ncan execute in 10 seconds. On my computer, without the new OP_INVOKE\ninstruction, it got through 1,089 batches. With this new optimization, it\nfinished 8,324 batches in the same time. That’s 7.6 times faster, which is a\nhuge improvement when it comes to programming language optimization.

\n

\n\"Bar\n

28 . 5 . 1Invoking fields

\n

The fundamental creed of optimization is: “Thou shalt not break correctness.”\nUsers like it when a language implementation gives\nthem an answer faster, but only if it’s the right answer. Alas, our\nimplementation of faster method invocations fails to uphold that principle:

\n
class Oops {\n  init() {\n    fun f() {\n      print "not a method";\n    }\n\n    this.field = f;\n  }\n}\n\nvar oops = Oops();\noops.field();\n
\n

The last line looks like a method call. The compiler thinks that it is and\ndutifully emits an OP_INVOKE instruction for it. However, it’s not. What is\nactually happening is a field access that returns a function which then gets\ncalled. Right now, instead of executing that correctly, our VM reports a runtime\nerror when it can’t find a method named “field”.

\n\n

Earlier, when we implemented OP_GET_PROPERTY, we handled both field and method\naccesses. To squash this new bug, we need to do the same thing for OP_INVOKE.

\n
  ObjInstance* instance = AS_INSTANCE(receiver);\n
vm.c
\nin invoke()
\n
\n\n  Value value;\n  if (tableGet(&instance->fields, name, &value)) {\n    vm.stackTop[-argCount - 1] = value;\n    return callValue(value, argCount);\n  }\n\n
  return invokeFromClass(instance->klass, name, argCount);\n
\n
vm.c, in invoke()
\n\n

Pretty simple fix. Before looking up a method on the instance’s class, we look\nfor a field with the same name. If we find a field, then we store it on the\nstack in place of the receiver, under the argument list. This is how\nOP_GET_PROPERTY behaves since the latter instruction executes before a\nsubsequent parenthesized list of arguments has been evaluated.

\n

Then we try to call that field’s value like the callable that it hopefully is.\nThe callValue() helper will check the value’s type and call it as appropriate\nor report a runtime error if the field’s value isn’t a callable type like a\nclosure.

\n

That’s all it takes to make our optimization fully safe. We do sacrifice a\nlittle performance, unfortunately. But that’s the price you have to pay\nsometimes. You occasionally get frustrated by optimizations you could do if\nonly the language wouldn’t allow some annoying corner case. But, as language\nimplementers, we have to play the game we’re given.

\n\n

The code we wrote here follows a typical pattern in optimization:

\n
    \n
  1. \n

    Recognize a common operation or sequence of operations that is performance\ncritical. In this case, it is a method access followed by a call.

    \n
    2. \n
  2. \n

    Add an optimized implementation of that pattern. That’s our OP_INVOKE\ninstruction.

    \n
    3. \n
  3. \n

    Guard the optimized code with some conditional logic that validates that the\npattern actually applies. If it does, stay on the fast path. Otherwise, fall\nback to a slower but more robust unoptimized behavior. Here, that means\nchecking that we are actually calling a method and not accessing a field.

    \n
    4. \n
\n

As your language work moves from getting the implementation working at all to\ngetting it to work faster, you will find yourself spending more and more\ntime looking for patterns like this and adding guarded optimizations for them.\nFull-time VM engineers spend much of their careers in this loop.

\n

But we can stop here for now. With this, clox now supports most of the features\nof an object-oriented programming language, and with respectable performance.

\n
\n

Challenges

\n
    \n
  1. \n

    The hash table lookup to find a class’s init() method is constant time,\nbut still fairly slow. Implement something faster. Write a benchmark and\nmeasure the performance difference.

    \n
    2. \n
  2. \n

    In a dynamically typed language like Lox, a single callsite may invoke a\nvariety of methods on a number of classes throughout a program’s execution.\nEven so, in practice, most of the time a callsite ends up calling the exact\nsame method on the exact same class for the duration of the run. Most calls\nare actually not polymorphic even if the language says they can be.

    \n

    How do advanced language implementations optimize based on that observation?

    \n
    3. \n
  3. \n

    When interpreting an OP_INVOKE instruction, the VM has to do two hash\ntable lookups. First, it looks for a field that could shadow a method, and\nonly if that fails does it look for a method. The former check is rarely\nuseful—most fields do not contain functions. But it is necessary\nbecause the language says fields and methods are accessed using the same\nsyntax, and fields shadow methods.

    \n

    That is a language choice that affects the performance of our\nimplementation. Was it the right choice? If Lox were your language, what\nwould you do?

    \n
    4. \n
\n
\n
\n

Design Note: Novelty Budget

\n

I still remember the first time I wrote a tiny BASIC program on a TRS-80 and\nmade a computer do something it hadn’t done before. It felt like a superpower.\nThe first time I cobbled together just enough of a parser and interpreter to let\nme write a tiny program in my own language that made a computer do a thing was\nlike some sort of higher-order meta-superpower. It was and remains a wonderful\nfeeling.

\n

I realized I could design a language that looked and behaved however I chose. It\nwas like I’d been going to a private school that required uniforms my whole life\nand then one day transferred to a public school where I could wear whatever I\nwanted. I don’t need to use curly braces for blocks? I can use something other\nthan an equals sign for assignment? I can do objects without classes? Multiple\ninheritance and multimethods? A dynamic language that overloads statically, by\narity?

\n

Naturally, I took that freedom and ran with it. I made the weirdest, most\narbitrary language design decisions. Apostrophes for generics. No commas between\narguments. Overload resolution that can fail at runtime. I did things\ndifferently just for difference’s sake.

\n

This is a very fun experience that I highly recommend. We need more weird,\navant-garde programming languages. I want to see more art languages. I still\nmake oddball toy languages for fun sometimes.

\n

However, if your goal is success where “success” is defined as a large number\nof users, then your priorities must be different. In that case, your primary\ngoal is to have your language loaded into the brains of as many people as\npossible. That’s really hard. It takes a lot of human effort to move a\nlanguage’s syntax and semantics from a computer into trillions of neurons.

\n

Programmers are naturally conservative with their time and cautious about what\nlanguages are worth uploading into their wetware. They don’t want to waste their\ntime on a language that ends up not being useful to them. As a language\ndesigner, your goal is thus to give them as much language power as you can with\nas little required learning as possible.

\n

One natural approach is simplicity. The fewer concepts and features your\nlanguage has, the less total volume of stuff there is to learn. This is one of\nthe reasons minimal scripting languages often find\nsuccess even though they aren’t as powerful as the big industrial languages—they are easier to get started with, and once they are in someone’s brain, the\nuser wants to keep using them.

\n\n

The problem with simplicity is that simply cutting features often sacrifices\npower and expressiveness. There is an art to finding features that punch above\ntheir weight, but often minimal languages simply do less.

\n

There is another path that avoids much of that problem. The trick is to realize\nthat a user doesn’t have to load your entire language into their head, just the\npart they don’t already have in there. As I mentioned in an earlier design\nnote, learning is about transferring the delta between what they\nalready know and what they need to know.

\n

Many potential users of your language already know some other programming\nlanguage. Any features your language shares with that language are essentially\n“free” when it comes to learning. It’s already in their head, they just have to\nrecognize that your language does the same thing.

\n

In other words, familiarity is another key tool to lower the adoption cost of\nyour language. Of course, if you fully maximize that attribute, the end result\nis a language that is completely identical to some existing one. That’s not a\nrecipe for success, because at that point there’s no incentive for users to\nswitch to your language at all.

\n

So you do need to provide some compelling differences. Some things your language\ncan do that other languages can’t, or at least can’t do as well. I believe this\nis one of the fundamental balancing acts of language design: similarity to other\nlanguages lowers learning cost, while divergence raises the compelling\nadvantages.

\n

I think of this balancing act in terms of a novelty\nbudget, or as Steve Klabnik calls it, a “strangeness budget”. Users\nhave a low threshold for the total amount of new stuff they are willing to\naccept to learn a new language. Exceed that, and they won’t show up.

\n\n

Anytime you add something new to your language that other languages don’t have,\nor anytime your language does something other languages do in a different way,\nyou spend some of that budget. That’s OK—you need to spend it to make your\nlanguage compelling. But your goal is to spend it wisely. For each feature or\ndifference, ask yourself how much compelling power it adds to your language and\nthen evaluate critically whether it pays its way. Is the change so valuable that\nit is worth blowing some of your novelty budget?

\n

In practice, I find this means that you end up being pretty conservative with\nsyntax and more adventurous with semantics. As fun as it is to put on a new\nchange of clothes, swapping out curly braces with some other block delimiter is\nvery unlikely to add much real power to the language, but it does spend some\nnovelty. It’s hard for syntax differences to carry their weight.

\n

On the other hand, new semantics can significantly increase the power of the\nlanguage. Multimethods, mixins, traits, reflection, dependent types, runtime\nmetaprogramming, etc. can radically level up what a user can do with the\nlanguage.

\n

Alas, being conservative like this is not as fun as just changing everything.\nBut it’s up to you to decide whether you want to chase mainstream success or not\nin the first place. We don’t all need to be radio-friendly pop bands. If you\nwant your language to be like free jazz or drone metal and are happy with the\nproportionally smaller (but likely more devoted) audience size, go for it.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/optimization.html", "content": "\n\n\n\nOptimization · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
30
\n

Optimization

\n\n
\n

The evening’s the best part of the day. You’ve done your day’s work. Now you\ncan put your feet up and enjoy it.

\n

Kazuo Ishiguro, The Remains of the Day

\n
\n

If I still lived in New Orleans, I’d call this chapter a lagniappe, a little\nsomething extra given for free to a customer. You’ve got a whole book and a\ncomplete virtual machine already, but I want you to have some more fun hacking\non clox. This time, we’re going for pure performance. We’ll apply two very\ndifferent optimizations to our virtual machine. In the process, you’ll get a\nfeel for measuring and improving the performance of a language implementation—or any program, really.

\n

30 . 1Measuring Performance

\n

Optimization means taking a working application and improving its\nperformance. An optimized program does the same thing, it just takes less\nresources to do so. The resource we usually think of when optimizing is runtime\nspeed, but it can also be important to reduce memory usage, startup time,\npersistent storage size, or network bandwidth. All physical resources have some\ncost—even if the cost is mostly in wasted human time—so optimization work\noften pays off.

\n

There was a time in the early days of computing that a skilled programmer could\nhold the entire hardware architecture and compiler pipeline in their head and\nunderstand a program’s performance just by thinking real hard. Those days are\nlong gone, separated from the present by microcode, cache lines, branch\nprediction, deep compiler pipelines, and mammoth instruction sets. We like to\npretend C is a “low-level” language, but the stack of technology between

\n
printf("Hello, world!");\n
\n

and a greeting appearing on screen is now perilously tall.

\n

Optimization today is an empirical science. Our program is a border collie\nsprinting through the hardware’s obstacle course. If we want her to reach the\nend faster, we can’t just sit and ruminate on canine physiology until\nenlightenment strikes. Instead, we need to observe her performance, see where\nshe stumbles, and then find faster paths for her to take.

\n

Much like agility training is particular to one dog and one obstacle course, we\ncan’t assume that our virtual machine optimizations will make all Lox programs\nrun faster on all hardware. Different Lox programs stress different areas of\nthe VM, and different architectures have their own strengths and weaknesses.

\n

30 . 1 . 1Benchmarks

\n

When we add new functionality, we validate correctness by writing tests—Lox\nprograms that use a feature and validate the VM’s behavior. Tests pin down\nsemantics and ensure we don’t break existing features when we add new ones. We\nhave similar needs when it comes to performance:

\n
    \n
  1. \n

    How do we validate that an optimization does improve performance, and by\nhow much?

    \n
    2. \n
  2. \n

    How do we ensure that other unrelated changes don’t regress performance?

    \n
    3. \n
\n

The Lox programs we write to accomplish those goals are benchmarks. These\nare carefully crafted programs that stress some part of the language\nimplementation. They measure not what the program does, but how long it takes to do it.

\n\n

By measuring the performance of a benchmark before and after a change, you can\nsee what your change does. When you land an optimization, all of the tests\nshould behave exactly the same as they did before, but hopefully the benchmarks\nrun faster.

\n

Once you have an entire suite of benchmarks, you can\nmeasure not just that an optimization changes performance, but on which\nkinds of code. Often you’ll find that some benchmarks get faster while others\nget slower. Then you have to make hard decisions about what kinds of code your\nlanguage implementation optimizes for.

\n

The suite of benchmarks you choose to write is a key part of that decision. In\nthe same way that your tests encode your choices around what correct behavior\nlooks like, your benchmarks are the embodiment of your priorities when it comes\nto performance. They will guide which optimizations you implement, so choose\nyour benchmarks carefully, and don’t forget to periodically reflect on whether\nthey are helping you reach your larger goals.

\n\n

Benchmarking is a subtle art. Like tests, you need to balance not overfitting to\nyour implementation while ensuring that the benchmark does actually tickle the\ncode paths that you care about. When you measure performance, you need to\ncompensate for variance caused by CPU throttling, caching, and other weird\nhardware and operating system quirks. I won’t give you a whole sermon here,\nbut treat benchmarking as its own skill that improves with practice.

\n

30 . 1 . 2Profiling

\n

OK, so you’ve got a few benchmarks now. You want to make them go faster. Now\nwhat? First of all, let’s assume you’ve done all the obvious, easy work. You are\nusing the right algorithms and data structures—or, at least, you aren’t using\nones that are aggressively wrong. I don’t consider using a hash table instead of\na linear search through a huge unsorted array “optimization” so much as “good\nsoftware engineering”.

\n

Since the hardware is too complex to reason about our program’s performance from\nfirst principles, we have to go out into the field. That means profiling. A\nprofiler, if you’ve never used one, is a tool that runs your program and tracks hardware resource use as the code\nexecutes. Simple ones show you how much time was spent in each function in your\nprogram. Sophisticated ones log data cache misses, instruction cache misses,\nbranch mispredictions, memory allocations, and all sorts of other metrics.

\n\n

There are many profilers out there for various operating systems and languages.\nOn whatever platform you program, it’s worth getting familiar with a decent\nprofiler. You don’t need to be a master. I have learned things within minutes of\nthrowing a program at a profiler that would have taken me days to discover on\nmy own through trial and error. Profilers are wonderful, magical tools.

\n

30 . 2Faster Hash Table Probing

\n

Enough pontificating, let’s get some performance charts going up and to the\nright. The first optimization we’ll do, it turns out, is about the tiniest\npossible change we could make to our VM.

\n

When I first got the bytecode virtual machine that clox is descended from\nworking, I did what any self-respecting VM hacker would do. I cobbled together a\ncouple of benchmarks, fired up a profiler, and ran those scripts through my\ninterpreter. In a dynamically typed language like Lox, a large fraction of user\ncode is field accesses and method calls, so one of my benchmarks looked\nsomething like this:

\n
class Zoo {\n  init() {\n    this.aardvark = 1;\n    this.baboon   = 1;\n    this.cat      = 1;\n    this.donkey   = 1;\n    this.elephant = 1;\n    this.fox      = 1;\n  }\n  ant()    { return this.aardvark; }\n  banana() { return this.baboon; }\n  tuna()   { return this.cat; }\n  hay()    { return this.donkey; }\n  grass()  { return this.elephant; }\n  mouse()  { return this.fox; }\n}\n\nvar zoo = Zoo();\nvar sum = 0;\nvar start = clock();\nwhile (sum < 100000000) {\n  sum = sum + zoo.ant()\n            + zoo.banana()\n            + zoo.tuna()\n            + zoo.hay()\n            + zoo.grass()\n            + zoo.mouse();\n}\n\nprint clock() - start;\nprint sum;\n
\n\n

If you’ve never seen a benchmark before, this might seem ludicrous. What is\ngoing on here? The program itself doesn’t intend to do\nanything useful. What it does do is call a bunch of methods and access a bunch\nof fields since those are the parts of the language we’re interested in. Fields\nand methods live in hash tables, so it takes care to populate at least a few interesting keys in those tables. That is all wrapped\nin a big loop to ensure our profiler has enough execution time to dig in and see\nwhere the cycles are going.

\n\n

Before I tell you what my profiler showed me, spend a minute taking a few\nguesses. Where in clox’s codebase do you think the VM spent most of its time? Is\nthere any code we’ve written in previous chapters that you suspect is\nparticularly slow?

\n

Here’s what I found: Naturally, the function with the greatest inclusive time is\nrun(). (Inclusive time means the total time spent in some function and all\nother functions it calls—the total time between when you enter the function\nand when it returns.) Since run() is the main bytecode execution loop, it\ndrives everything.

\n

Inside run(), there are small chunks of time sprinkled in various cases in the\nbytecode switch for common instructions like OP_POP, OP_RETURN, and\nOP_ADD. The big heavy instructions are OP_GET_GLOBAL with 17% of the\nexecution time, OP_GET_PROPERTY at 12%, and OP_INVOKE which takes a whopping\n42% of the total running time.

\n

So we’ve got three hotspots to optimize? Actually, no. Because it turns out\nthose three instructions spend almost all of their time inside calls to the same\nfunction: tableGet(). That function claims a whole 72% of the execution time\n(again, inclusive). Now, in a dynamically typed language, we expect to spend a\nfair bit of time looking stuff up in hash tables—it’s sort of the price of\ndynamism. But, still, wow.

\n

30 . 2 . 1Slow key wrapping

\n

If you take a look at tableGet(), you’ll see it’s mostly a wrapper around a\ncall to findEntry() where the actual hash table lookup happens. To refresh\nyour memory, here it is in full:

\n
static Entry* findEntry(Entry* entries, int capacity,\n                        ObjString* key) {\n  uint32_t index = key->hash % capacity;\n  Entry* tombstone = NULL;\n\n  for (;;) {\n    Entry* entry = &entries[index];\n    if (entry->key == NULL) {\n      if (IS_NIL(entry->value)) {\n        // Empty entry.\n        return tombstone != NULL ? tombstone : entry;\n      } else {\n        // We found a tombstone.\n        if (tombstone == NULL) tombstone = entry;\n      }\n    } else if (entry->key == key) {\n      // We found the key.\n      return entry;\n    }\n\n    index = (index + 1) % capacity;\n  }\n}\n
\n

When running that previous benchmark—on my machine, at least—the VM spends\n70% of the total execution time on one line in this function. Any guesses as\nto which one? No? It’s this:

\n
  uint32_t index = key->hash % capacity;\n
\n

That pointer dereference isn’t the problem. It’s the little %. It turns out\nthe modulo operator is really slow. Much slower than other arithmetic operators. Can we do something better?

\n\n

In the general case, it’s really hard to re-implement a fundamental arithmetic\noperator in user code in a way that’s faster than what the CPU itself can do.\nAfter all, our C code ultimately compiles down to the CPU’s own arithmetic\noperations. If there were tricks we could use to go faster, the chip would\nalready be using them.

\n

However, we can take advantage of the fact that we know more about our problem\nthan the CPU does. We use modulo here to take a key string’s hash code and\nwrap it to fit within the bounds of the table’s entry array. That array starts\nout at eight elements and grows by a factor of two each time. We know—and the\nCPU and C compiler do not—that our table’s size is always a power of two.

\n

Because we’re clever bit twiddlers, we know a faster way to calculate the\nremainder of a number modulo a power of two: bit masking. Let’s say we want\nto calculate 229 modulo 64. The answer is 37, which is not particularly apparent\nin decimal, but is clearer when you view those numbers in binary:

\"The\n

On the left side of the illustration, notice how the result (37) is simply the\ndividend (229) with the highest two bits shaved off? Those two highest bits are\nthe bits at or to the left of the divisor’s single 1 bit.

\n

On the right side, we get the same result by taking 229 and bitwise AND-ing it with 63, which is one less than our\noriginal power of two divisor. Subtracting one from a power of two gives you a\nseries of 1 bits. That is exactly the mask we need in order to strip out those\ntwo leftmost bits.

\n

In other words, you can calculate a number modulo any power of two by simply\nAND-ing it with that power of two minus one. I’m\nnot enough of a mathematician to prove to you that this works, but if you\nthink it through, it should make sense. We can replace that slow modulo operator\nwith a very fast decrement and bitwise AND. We\nsimply change the offending line of code to this:

\n
static Entry* findEntry(Entry* entries, int capacity,\n                        ObjString* key) {\n
table.c
\nin findEntry()
\nreplace 1 line
\n
  uint32_t index = key->hash & (capacity - 1);\n
  Entry* tombstone = NULL;\n
\n
table.c, in findEntry(), replace 1 line
\n\n

CPUs love bitwise operators, so it’s hard to improve on that.

\n\n

Our linear probing search may need to wrap around the end of the array, so there\nis another modulo in findEntry() to update.

\n
      // We found the key.\n      return entry;\n    }\n\n
table.c
\nin findEntry()
\nreplace 1 line
\n
    index = (index + 1) & (capacity - 1);\n
  }\n
\n
table.c, in findEntry(), replace 1 line
\n\n

This line didn’t show up in the profiler since most searches don’t wrap.

\n

The findEntry() function has a sister function, tableFindString() that does\na hash table lookup for interning strings. We may as well apply the same\noptimizations there too. This function is called only when interning strings,\nwhich wasn’t heavily stressed by our benchmark. But a Lox program that created\nlots of strings might noticeably benefit from this change.

\n
  if (table->count == 0) return NULL;\n\n
table.c
\nin tableFindString()
\nreplace 1 line
\n
  uint32_t index = hash & (table->capacity - 1);\n
  for (;;) {\n    Entry* entry = &table->entries[index];\n
\n
table.c, in tableFindString(), replace 1 line
\n\n

And also when the linear probing wraps around.

\n
      return entry->key;\n    }\n\n
table.c
\nin tableFindString()
\nreplace 1 line
\n
    index = (index + 1) & (table->capacity - 1);\n
  }\n
\n
table.c, in tableFindString(), replace 1 line
\n\n

Let’s see if our fixes were worth it. I tweaked that zoological benchmark to\ncount how many batches of 10,000 calls it can run in\nten seconds. More batches equals faster performance. On my machine using the\nunoptimized code, the benchmark gets through 3,192 batches. After this\noptimization, that jumps to 6,249.

\"Bar\n

That’s almost exactly twice as much work in the same amount of time. We made the\nVM twice as fast (usual caveat: on this benchmark). That is a massive win when\nit comes to optimization. Usually you feel good if you can claw a few percentage\npoints here or there. Since methods, fields, and global variables are so\nprevalent in Lox programs, this tiny optimization improves performance across\nthe board. Almost every Lox program benefits.

\n\n

Now, the point of this section is not that the modulo operator is profoundly\nevil and you should stamp it out of every program you ever write. Nor is it that\nmicro-optimization is a vital engineering skill. It’s rare that a performance\nproblem has such a narrow, effective solution. We got lucky.

\n

The point is that we didn’t know that the modulo operator was a performance\ndrain until our profiler told us so. If we had wandered around our VM’s codebase\nblindly guessing at hotspots, we likely wouldn’t have noticed it. What I want\nyou to take away from this is how important it is to have a profiler in your\ntoolbox.

\n

To reinforce that point, let’s go ahead and run the original benchmark in our\nnow-optimized VM and see what the profiler shows us. On my machine, tableGet()\nis still a fairly large chunk of execution time. That’s to be expected for a\ndynamically typed language. But it has dropped from 72% of the total execution\ntime down to 35%. That’s much more in line with what we’d like to see and shows\nthat our optimization didn’t just make the program faster, but made it faster\nin the way we expected. Profilers are as useful for verifying solutions as\nthey are for discovering problems.

\n

30 . 3NaN Boxing

\n

This next optimization has a very different feel. Thankfully, despite the odd\nname, it does not involve punching your grandmother. It’s different, but not,\nlike, that different. With our previous optimization, the profiler told us\nwhere the problem was, and we merely had to use some ingenuity to come up with a\nsolution.

\n

This optimization is more subtle, and its performance effects more scattered\nacross the virtual machine. The profiler won’t help us come up with this.\nInstead, it was invented by someone thinking deeply\nabout the lowest levels of machine architecture.

\n\n

Like the heading says, this optimization is called NaN boxing or sometimes\nNaN tagging. Personally I like the latter name because “boxing” tends to imply\nsome kind of heap-allocated representation, but the former seems to be the more\nwidely used term. This technique changes how we represent values in the VM.

\n

On a 64-bit machine, our Value type takes up 16 bytes. The struct has two\nfields, a type tag and a union for the payload. The largest fields in the union\nare an Obj pointer and a double, which are both 8 bytes. To keep the union field\naligned to an 8-byte boundary, the compiler adds padding after the tag too:

\"Byte\n

That’s pretty big. If we could cut that down, then the VM could pack more values\ninto the same amount of memory. Most computers have plenty of RAM these days, so\nthe direct memory savings aren’t a huge deal. But a smaller representation means\nmore Values fit in a cache line. That means fewer cache misses, which affects\nspeed.

\n

If Values need to be aligned to their largest payload size, and a Lox number or\nObj pointer needs a full 8 bytes, how can we get any smaller? In a dynamically\ntyped language like Lox, each value needs to carry not just its payload, but\nenough additional information to determine the value’s type at runtime. If a Lox\nnumber is already using the full 8 bytes, where could we squirrel away a couple\nof extra bits to tell the runtime “this is a number”?

\n

This is one of the perennial problems for dynamic language hackers. It\nparticularly bugs them because statically typed languages don’t generally have\nthis problem. The type of each value is known at compile time, so no extra\nmemory is needed at runtime to track it. When your C compiler compiles a 32-bit\nint, the resulting variable gets exactly 32 bits of storage.

\n

Dynamic language folks hate losing ground to the static camp, so they’ve come up\nwith a number of very clever ways to pack type information and a payload into a\nsmall number of bits. NaN boxing is one of those. It’s a particularly good fit\nfor languages like JavaScript and Lua, where all numbers are double-precision\nfloating point. Lox is in that same boat.

\n

30 . 3 . 1What is (and is not) a number?

\n

Before we start optimizing, we need to really understand how our friend the CPU\nrepresents floating-point numbers. Almost all machines today use the same\nscheme, encoded in the venerable scroll IEEE 754, known to mortals as the\n“IEEE Standard for Floating-Point Arithmetic”.

\n

In the eyes of your computer, a 64-bit,\ndouble-precision, IEEE floating-point number looks like this:

\n\"Bit\n
    \n
  • \n

    Starting from the right, the first 52 bits are the fraction,\nmantissa, or significand bits. They represent the significant digits\nof the number, as a binary integer.

    \n
    • \n
  • \n

    Next to that are 11 exponent bits. These tell you how far the mantissa\nis shifted away from the decimal (well, binary) point.

    \n
    • \n
  • \n

    The highest bit is the sign bit, which\nindicates whether the number is positive or negative.

    \n
    • \n
\n

I know that’s a little vague, but this chapter isn’t a deep dive on\nfloating point representation. If you want to know how the exponent and mantissa\nplay together, there are already better explanations out there than I could\nwrite.

\n\n

The important part for our purposes is that the spec carves out a special case\nexponent. When all of the exponent bits are set, then instead of just\nrepresenting a really big number, the value has a different meaning. These\nvalues are “Not a Number” (hence, NaN) values. They represent concepts like\ninfinity or the result of division by zero.

\n

Any double whose exponent bits are all set is a NaN, regardless of the\nmantissa bits. That means there’s lots and lots of different NaN bit patterns.\nIEEE 754 divides those into two categories. Values where the highest mantissa\nbit is 0 are called signalling NaNs, and the others are quiet NaNs.\nSignalling NaNs are intended to be the result of erroneous computations, like\ndivision by zero. A chip may detect when one of these\nvalues is produced and abort a program completely. They may self-destruct if you\ntry to read one.

\n\n

Quiet NaNs are supposed to be safer to use. They don’t represent useful numeric\nvalues, but they should at least not set your hand on fire if you touch them.

\n

Every double with all of its exponent bits set and its highest mantissa bit set\nis a quiet NaN. That leaves 52 bits unaccounted for. We’ll avoid one of those so\nthat we don’t step on Intel’s “QNaN Floating-Point Indefinite” value, leaving us\n51 bits. Those remaining bits can be anything. We’re talking\n2,251,799,813,685,248 unique quiet NaN bit patterns.

\"The\n

This means a 64-bit double has enough room to store all of the various different\nnumeric floating-point values and also has room for another 51 bits of data\nthat we can use however we want. That’s plenty of room to set aside a couple of\nbit patterns to represent Lox’s nil, true, and false values. But what\nabout Obj pointers? Don’t pointers need a full 64 bits too?

\n

Fortunately, we have another trick up our other sleeve. Yes, technically\npointers on a 64-bit architecture are 64 bits. But, no architecture I know of\nactually uses that entire address space. Instead, most widely used chips today\nonly ever use the low 48 bits. The remaining 16 bits are\neither unspecified or always zero.

\n\n

If we’ve got 51 bits, we can stuff a 48-bit pointer in there with three bits to\nspare. Those three bits are just enough to store tiny type tags to distinguish\nbetween nil, Booleans, and Obj pointers.

\n

That’s NaN boxing. Within a single 64-bit double, you can store all of the\ndifferent floating-point numeric values, a pointer, or any of a couple of other\nspecial sentinel values. Half the memory usage of our current Value struct,\nwhile retaining all of the fidelity.

\n

What’s particularly nice about this representation is that there is no need to\nconvert a numeric double value into a “boxed” form. Lox numbers are just\nnormal, 64-bit doubles. We still need to check their type before we use them,\nsince Lox is dynamically typed, but we don’t need to do any bit shifting or\npointer indirection to go from “value” to “number”.

\n

For the other value types, there is a conversion step, of course. But,\nfortunately, our VM hides all of the mechanism to go from values to raw types\nbehind a handful of macros. Rewrite those to implement NaN boxing, and the rest\nof the VM should just work.

\n

30 . 3 . 2Conditional support

\n

I know the details of this new representation aren’t clear in your head yet.\nDon’t worry, they will crystallize as we work through the implementation. Before\nwe get to that, we’re going to put some compile-time scaffolding in place.

\n

For our previous optimization, we rewrote the previous slow code and called it\ndone. This one is a little different. NaN boxing relies on some very low-level\ndetails of how a chip represents floating-point numbers and pointers. It\nprobably works on most CPUs you’re likely to encounter, but you can never be\ntotally sure.

\n

It would suck if our VM completely lost support for an architecture just because\nof its value representation. To avoid that, we’ll maintain support for both\nthe old tagged union implementation of Value and the new NaN-boxed form. We\nselect which representation we want at compile time using this flag:

\n
#include <stdint.h>\n\n
common.h
\n
#define NAN_BOXING\n
#define DEBUG_PRINT_CODE\n
\n
common.h
\n\n

If that’s defined, the VM uses the new form. Otherwise, it reverts to the old\nstyle. The few pieces of code that care about the details of the value\nrepresentation—mainly the handful of macros for wrapping and unwrapping\nValues—vary based on whether this flag is set. The rest of the VM can\ncontinue along its merry way.

\n

Most of the work happens in the “value” module where we add a section for the\nnew type.

\n
typedef struct ObjString ObjString;\n\n
value.h
\n
#ifdef NAN_BOXING\n\ntypedef uint64_t Value;\n\n#else\n\n
typedef enum {\n
\n
value.h
\n\n

When NaN boxing is enabled, the actual type of a Value is a flat, unsigned\n64-bit integer. We could use double instead, which would make the macros for\ndealing with Lox numbers a little simpler. But all of the other macros need to\ndo bitwise operations and uint64_t is a much friendlier type for that. Outside\nof this module, the rest of the VM doesn’t really care one way or the other.

\n

Before we start re-implementing those macros, we close the #else branch of the\n#ifdef at the end of the definitions for the old representation.

\n
#define OBJ_VAL(object)   ((Value){VAL_OBJ, {.obj = (Obj*)object}})\n
value.h
\n
\n\n#endif\n
\n\ntypedef struct {\n
\n
value.h
\n\n

Our remaining task is simply to fill in that first #ifdef section with new\nimplementations of all the stuff already in the #else side. We’ll work through\nit one value type at a time, from easiest to hardest.

\n

30 . 3 . 3Numbers

\n

We’ll start with numbers since they have the most direct representation under\nNaN boxing. To “convert” a C double to a NaN-boxed clox Value, we don’t need to\ntouch a single bit—the representation is exactly the same. But we do need to\nconvince our C compiler of that fact, which we made harder by defining Value to\nbe uint64_t.

\n

We need to get the compiler to take a set of bits that it thinks are a double\nand use those same bits as a uint64_t, or vice versa. This is called type\npunning. C and C++ programmers have been doing this since the days of bell\nbottoms and 8-tracks, but the language specifications have hesitated to say which of the many ways to do this is\nofficially sanctioned.

\n\n

I know one way to convert a double to Value and back that I believe is\nsupported by both the C and C++ specs. Unfortunately, it doesn’t fit in a single\nexpression, so the conversion macros have to call out to helper functions.\nHere’s the first macro:

\n
typedef uint64_t Value;\n
value.h
\n
\n\n#define NUMBER_VAL(num) numToValue(num)\n
\n\n#else\n
\n
value.h
\n\n

That macro passes the double here:

\n
#define NUMBER_VAL(num) numToValue(num)\n
value.h
\n
\n\nstatic inline Value numToValue(double num) {\n  Value value;\n  memcpy(&value, &num, sizeof(double));\n  return value;\n}\n
\n\n#else\n
\n
value.h
\n\n

I know, weird, right? The way to treat a series of bytes as having a different\ntype without changing their value at all is memcpy()? This looks horrendously\nslow: Create a local variable. Pass its address to the operating system through\na syscall to copy a few bytes. Then return the result, which is the exact same\nbytes as the input. Thankfully, because this is the supported idiom for type\npunning, most compilers recognize the pattern and optimize away the memcpy()\nentirely.

\n

“Unwrapping” a Lox number is the mirror image.

\n
typedef uint64_t Value;\n
value.h
\n
\n\n#define AS_NUMBER(value)    valueToNum(value)\n
\n\n#define NUMBER_VAL(num) numToValue(num)\n
\n
value.h
\n\n

That macro calls this function:

\n
#define NUMBER_VAL(num) numToValue(num)\n
value.h
\n
\n\nstatic inline double valueToNum(Value value) {\n  double num;\n  memcpy(&num, &value, sizeof(Value));\n  return num;\n}\n
\n\nstatic inline Value numToValue(double num) {\n
\n
value.h
\n\n

It works exactly the same except we swap the types. Again, the compiler will\neliminate all of it. Even though those calls to\nmemcpy() will disappear, we still need to show the compiler which memcpy()\nwe’re calling so we also need an include.

\n\n
#define clox_value_h\n
value.h
\n
\n\n#include <string.h>\n
\n\n#include "common.h"\n
\n
value.h
\n\n

That was a lot of code to ultimately do nothing but silence the C type checker.\nDoing a runtime type test on a Lox number is a little more interesting. If all\nwe have are exactly the bits for a double, how do we tell that it is a double?\nIt’s time to get bit twiddling.

\n
typedef uint64_t Value;\n
value.h
\n
\n\n#define IS_NUMBER(value)    (((value) & QNAN) != QNAN)\n
\n\n#define AS_NUMBER(value)    valueToNum(value)\n
\n
value.h
\n\n

We know that every Value that is not a number will use a special quiet NaN\nrepresentation. And we presume we have correctly avoided any of the meaningful\nNaN representations that may actually be produced by doing arithmetic on\nnumbers.

\n

If the double has all of its NaN bits set, and the quiet NaN bit set, and one\nmore for good measure, we can be pretty certain it\nis one of the bit patterns we ourselves have set aside for other types. To check\nthat, we mask out all of the bits except for our set of quiet NaN bits. If all\nof those bits are set, it must be a NaN-boxed value of some other Lox type.\nOtherwise, it is actually a number.

\n\n

The set of quiet NaN bits are declared like this:

\n
#ifdef NAN_BOXING\n
value.h
\n
\n\n#define QNAN     ((uint64_t)0x7ffc000000000000)\n
\n\ntypedef uint64_t Value;\n
\n
value.h
\n\n

It would be nice if C supported binary literals. But if you do the conversion,\nyou’ll see that value is the same as this:

\"The\n

This is exactly all of the exponent bits, plus the quiet NaN bit, plus one extra\nto dodge that Intel value.

\n

30 . 3 . 4Nil, true, and false

\n

The next type to handle is nil. That’s pretty simple since there’s only one\nnil value and thus we need only a single bit pattern to represent it. There\nare two other singleton values, the two Booleans, true and false. This calls\nfor three total unique bit patterns.

\n

Two bits give us four different combinations, which is plenty. We claim the two\nlowest bits of our unused mantissa space as a “type tag” to determine which of\nthese three singleton values we’re looking at. The three type tags are defined\nlike so:

\n
#define QNAN     ((uint64_t)0x7ffc000000000000)\n
value.h
\n
\n\n#define TAG_NIL   1 // 01.\n#define TAG_FALSE 2 // 10.\n#define TAG_TRUE  3 // 11.\n
\n\ntypedef uint64_t Value;\n
\n
value.h
\n\n

Our representation of nil is thus all of the bits required to define our\nquiet NaN representation along with the nil type tag bits:

\"The\n

In code, we check the bits like so:

\n
#define AS_NUMBER(value)    valueToNum(value)\n\n
value.h
\n
#define NIL_VAL         ((Value)(uint64_t)(QNAN | TAG_NIL))\n
#define NUMBER_VAL(num) numToValue(num)\n
\n
value.h
\n\n

We simply bitwise OR the quiet NaN bits and the\ntype tag, and then do a little cast dance to teach the C compiler what we want\nthose bits to mean.

\n

Since nil has only a single bit representation, we can use equality on\nuint64_t to see if a Value is nil.

\n

\n
typedef uint64_t Value;\n\n
value.h
\n
#define IS_NIL(value)       ((value) == NIL_VAL)\n
#define IS_NUMBER(value)    (((value) & QNAN) != QNAN)\n
\n
value.h
\n\n

You can guess how we define the true and false values.

\n
#define AS_NUMBER(value)    valueToNum(value)\n\n
value.h
\n
#define FALSE_VAL       ((Value)(uint64_t)(QNAN | TAG_FALSE))\n#define TRUE_VAL        ((Value)(uint64_t)(QNAN | TAG_TRUE))\n
#define NIL_VAL         ((Value)(uint64_t)(QNAN | TAG_NIL))\n
\n
value.h
\n\n

The bits look like this:

\"The\n

To convert a C bool into a Lox Boolean, we rely on these two singleton values\nand the good old conditional operator.

\n
#define AS_NUMBER(value)    valueToNum(value)\n\n
value.h
\n
#define BOOL_VAL(b)     ((b) ? TRUE_VAL : FALSE_VAL)\n
#define FALSE_VAL       ((Value)(uint64_t)(QNAN | TAG_FALSE))\n
\n
value.h
\n\n

There’s probably a cleverer bitwise way to do this, but my hunch is that the\ncompiler can figure one out faster than I can. Going the other direction is\nsimpler.

\n
#define IS_NUMBER(value)    (((value) & QNAN) != QNAN)\n\n
value.h
\n
#define AS_BOOL(value)      ((value) == TRUE_VAL)\n
#define AS_NUMBER(value)    valueToNum(value)\n
\n
value.h
\n\n

Since we know there are exactly two Boolean bit representations in Lox—unlike\nin C where any non-zero value can be considered “true”—if it ain’t true, it\nmust be false. This macro does assume you call it only on a Value that you\nknow is a Lox Boolean. To check that, there’s one more macro.

\n
typedef uint64_t Value;\n\n
value.h
\n
#define IS_BOOL(value)      (((value) | 1) == TRUE_VAL)\n
#define IS_NIL(value)       ((value) == NIL_VAL)\n
\n
value.h
\n\n

That looks a little strange. A more obvious macro would look like this:

\n
#define IS_BOOL(v) ((v) == TRUE_VAL || (v) == FALSE_VAL)\n
\n

Unfortunately, that’s not safe. The expansion mentions v twice, which means if\nthat expression has any side effects, they will be executed twice. We could have\nthe macro call out to a separate function, but, ugh, what a chore.

\n

Instead, we bitwise OR a 1 onto the value to\nmerge the only two valid Boolean bit patterns. That leaves three potential\nstates the value can be in:

\n
    \n
  1. \n

    It was FALSE_VAL and has now been converted to TRUE_VAL.

    \n
    2. \n
  2. \n

    It was TRUE_VAL and the | 1 did nothing and it’s still TRUE_VAL.

    \n
    3. \n
  3. \n

    It’s some other, non-Boolean value.

    \n
    4. \n
\n

At that point, we can simply compare the result to TRUE_VAL to see if we’re\nin the first two states or the third.

\n

30 . 3 . 5Objects

\n

The last value type is the hardest. Unlike the singleton values, there are\nbillions of different pointer values we need to box inside a NaN. This means we\nneed both some kind of tag to indicate that these particular NaNs are Obj\npointers, and room for the addresses themselves.

\n

The tag bits we used for the singleton values are in the region where I decided\nto store the pointer itself, so we can’t easily use a different bit there to indicate that the value is an object reference.\nHowever, there is another bit we aren’t using. Since all our NaN values are not\nnumbers—it’s right there in the name—the sign bit isn’t used for anything.\nWe’ll go ahead and use that as the type tag for objects. If one of our quiet\nNaNs has its sign bit set, then it’s an Obj pointer. Otherwise, it must be one\nof the previous singleton values.

\n\n

If the sign bit is set, then the remaining low bits store the pointer to the\nObj:

\"Bit\n

To convert a raw Obj pointer to a Value, we take the pointer and set all of the\nquiet NaN bits and the sign bit.

\n
#define NUMBER_VAL(num) numToValue(num)\n
value.h
\n
#define OBJ_VAL(obj) \\\n    (Value)(SIGN_BIT | QNAN | (uint64_t)(uintptr_t)(obj))\n
\n\nstatic inline double valueToNum(Value value) {\n
\n
value.h
\n\n

The pointer itself is a full 64 bits, and in principle,\nit could thus overlap with some of those quiet NaN and sign bits. But in\npractice, at least on the architectures I’ve tested, everything above the 48th\nbit in a pointer is always zero. There’s a lot of casting going on here, which\nI’ve found is necessary to satisfy some of the pickiest C compilers, but the\nend result is just jamming some bits together.

\n\n

We define the sign bit like so:

\n
#ifdef NAN_BOXING\n\n
value.h
\n
#define SIGN_BIT ((uint64_t)0x8000000000000000)\n
#define QNAN     ((uint64_t)0x7ffc000000000000)\n\n
\n
value.h
\n\n

To get the Obj pointer back out, we simply mask off all of those extra bits.

\n
#define AS_NUMBER(value)    valueToNum(value)\n
value.h
\n
#define AS_OBJ(value) \\\n    ((Obj*)(uintptr_t)((value) & ~(SIGN_BIT | QNAN)))\n
\n\n#define BOOL_VAL(b)     ((b) ? TRUE_VAL : FALSE_VAL)\n
\n
value.h
\n\n

The tilde (~), if you haven’t done enough bit manipulation to encounter it\nbefore, is bitwise NOT. It toggles all ones and\nzeroes in its operand. By masking the value with the bitwise negation of the\nquiet NaN and sign bits, we clear those bits and let the pointer bits remain.

\n

One last macro:

\n
#define IS_NUMBER(value)    (((value) & QNAN) != QNAN)\n
value.h
\n
#define IS_OBJ(value) \\\n    (((value) & (QNAN | SIGN_BIT)) == (QNAN | SIGN_BIT))\n
\n\n#define AS_BOOL(value)      ((value) == TRUE_VAL)\n
\n
value.h
\n\n

A Value storing an Obj pointer has its sign bit set, but so does any negative\nnumber. To tell if a Value is an Obj pointer, we need to check that both the\nsign bit and all of the quiet NaN bits are set. This is similar to how we detect\nthe type of the singleton values, except this time we use the sign bit as the\ntag.

\n

30 . 3 . 6Value functions

\n

The rest of the VM usually goes through the macros when working with Values, so\nwe are almost done. However, there are a couple of functions in the “value”\nmodule that peek inside the otherwise black box of Value and work with its\nencoding directly. We need to fix those too.

\n

The first is printValue(). It has separate code for each value type. We no\nlonger have an explicit type enum we can switch on, so instead we use a series\nof type tests to handle each kind of value.

\n
void printValue(Value value) {\n
value.c
\nin printValue()
\n
#ifdef NAN_BOXING\n  if (IS_BOOL(value)) {\n    printf(AS_BOOL(value) ? "true" : "false");\n  } else if (IS_NIL(value)) {\n    printf("nil");\n  } else if (IS_NUMBER(value)) {\n    printf("%g", AS_NUMBER(value));\n  } else if (IS_OBJ(value)) {\n    printObject(value);\n  }\n#else\n
  switch (value.type) {\n
\n
value.c, in printValue()
\n\n

This is technically a tiny bit slower than a switch, but compared to the\noverhead of actually writing to a stream, it’s negligible.

\n

We still support the original tagged union representation, so we keep the old\ncode and enclose it in the #else conditional section.

\n
  }\n
value.c
\nin printValue()
\n
#endif\n
}\n
\n
value.c, in printValue()
\n\n

The other operation is testing two values for equality.

\n
bool valuesEqual(Value a, Value b) {\n
value.c
\nin valuesEqual()
\n
#ifdef NAN_BOXING\n  return a == b;\n#else\n
  if (a.type != b.type) return false;\n
\n
value.c, in valuesEqual()
\n\n

It doesn’t get much simpler than that! If the two bit representations are\nidentical, the values are equal. That does the right thing for the singleton\nvalues since each has a unique bit representation and they are only equal to\nthemselves. It also does the right thing for Obj pointers, since objects use\nidentity for equality—two Obj references are equal only if they point to the\nexact same object.

\n

It’s mostly correct for numbers too. Most floating-point numbers with\ndifferent bit representations are distinct numeric values. Alas, IEEE 754\ncontains a pothole to trip us up. For reasons that aren’t entirely clear to me,\nthe spec mandates that NaN values are not equal to themselves. This isn’t a\nproblem for the special quiet NaNs that we are using for our own purposes. But\nit’s possible to produce a “real” arithmetic NaN in Lox, and if we want to\ncorrectly implement IEEE 754 numbers, then the resulting value is not supposed\nto be equal to itself. More concretely:

\n
var nan = 0/0;\nprint nan == nan;\n
\n

IEEE 754 says this program is supposed to print “false”. It does the right thing\nwith our old tagged union representation because the VAL_NUMBER case applies\n== to two values that the C compiler knows are doubles. Thus the compiler\ngenerates the right CPU instruction to perform an IEEE floating-point equality.

\n

Our new representation breaks that by defining Value to be a uint64_t. If we\nwant to be fully compliant with IEEE 754, we need to handle this case.

\n
#ifdef NAN_BOXING\n
value.c
\nin valuesEqual()
\n
  if (IS_NUMBER(a) && IS_NUMBER(b)) {\n    return AS_NUMBER(a) == AS_NUMBER(b);\n  }\n
  return a == b;\n
\n
value.c, in valuesEqual()
\n\n

I know, it’s weird. And there is a performance cost to doing this type test\nevery time we check two Lox values for equality. If we are willing to sacrifice\na little compatibility—who really cares if NaN is\nnot equal to itself?—we could leave this off. I’ll leave it up to you to\ndecide how pedantic you want to be.

\n\n

Finally, we close the conditional compilation section around the old\nimplementation.

\n
  }\n
value.c
\nin valuesEqual()
\n
#endif\n
}\n
\n
value.c, in valuesEqual()
\n\n

And that’s it. This optimization is complete, as is our clox virtual machine.\nThat was the last line of new code in the book.

\n

30 . 3 . 7Evaluating performance

\n

The code is done, but we still need to figure out if we actually made anything\nbetter with these changes. Evaluating an optimization like this is very\ndifferent from the previous one. There, we had a clear hotspot visible in the\nprofiler. We fixed that part of the code and could instantly see the hotspot\nget faster.

\n

The effects of changing the value representation are more diffused. The macros\nare expanded in place wherever they are used, so the performance changes are\nspread across the codebase in a way that’s hard for many profilers to track\nwell, especially in an optimized build.

\n\n

We also can’t easily reason about the effects of our change. We’ve made values\nsmaller, which reduces cache misses all across the VM. But the actual real-world\nperformance effect of that change is highly dependent on the memory use of the\nLox program being run. A tiny Lox microbenchmark may not have enough values\nscattered around in memory for the effect to be noticeable, and even things like\nthe addresses handed out to us by the C memory allocator can impact the results.

\n

If we did our job right, basically everything gets a little faster, especially\non larger, more complex Lox programs. But it is possible that the extra bitwise\noperations we do when NaN-boxing values nullify the gains from the better\nmemory use. Doing performance work like this is unnerving because you can’t\neasily prove that you’ve made the VM better. You can’t point to a single\nsurgically targeted microbenchmark and say, “There, see?”

\n

Instead, what we really need is a suite of larger benchmarks. Ideally, they\nwould be distilled from real-world applications—not that such a thing exists\nfor a toy language like Lox. Then we can measure the aggregate performance\nchanges across all of those. I did my best to cobble together a handful of\nlarger Lox programs. On my machine, the new value representation seems to make\neverything roughly 10% faster across the board.

\n

That’s not a huge improvement, especially compared to the profound effect of\nmaking hash table lookups faster. I added this optimization in large part\nbecause it’s a good example of a certain kind of performance work you may\nexperience, and honestly, because I think it’s technically really cool. It might\nnot be the first thing I would reach for if I were seriously trying to make clox\nfaster. There is probably other, lower-hanging fruit.

\n

But, if you find yourself working on a program where all of the easy wins have\nbeen taken, then at some point you may want to think about tuning your value\nrepresentation. I hope this chapter has shined a light on some of the options\nyou have in that area.

\n

30 . 4Where to Next

\n

We’ll stop here with the Lox language and our two interpreters. We could tinker\non it forever, adding new language features and clever speed improvements. But,\nfor this book, I think we’ve reached a natural place to call our work complete.\nI won’t rehash everything we’ve learned in the past many pages. You were there\nwith me and you remember. Instead, I’d like to take a minute to talk about where\nyou might go from here. What is the next step in your programming language\njourney?

\n

Most of you probably won’t spend a significant part of your career working in\ncompilers or interpreters. It’s a pretty small slice of the computer science\nacademia pie, and an even smaller segment of software engineering in industry.\nThat’s OK. Even if you never work on a compiler again in your life, you will\ncertainly use one, and I hope this book has equipped you with a better\nunderstanding of how the programming languages you use are designed and\nimplemented.

\n

You have also learned a handful of important, fundamental data structures and\ngotten some practice doing low-level profiling and optimization work. That kind\nof expertise is helpful no matter what domain you program in.

\n

I also hope I gave you a new way of looking at and\nsolving problems. Even if you never work on a language again, you may be\nsurprised to discover how many programming problems can be seen as\nlanguage-like. Maybe that report generator you need to write can be modeled as\na series of stack-based “instructions” that the generator “executes”. That user\ninterface you need to render looks an awful lot like traversing an AST.

\n\n

If you do want to go further down the programming language rabbit hole, here\nare some suggestions for which branches in the tunnel to explore:

\n
    \n
  • \n

    Our simple, single-pass bytecode compiler pushed us towards mostly runtime\noptimization. In a mature language implementation, compile-time optimization\nis generally more important, and the field of compiler optimizations is\nincredibly rich. Grab a classic compilers book,\nand rebuild the front end of clox or jlox to be a sophisticated compilation\npipeline with some interesting intermediate representations and optimization\npasses.

    \n

    Dynamic typing will place some restrictions on how far you can go, but there\nis still a lot you can do. Or maybe you want to take a big leap and add\nstatic types and a type checker to Lox. That will certainly give your front\nend a lot more to chew on.

    \n
    • \n
  • \n

    In this book, I aim to be correct, but not particularly rigorous. My goal is\nmostly to give you an intuition and a feel for doing language work. If you\nlike more precision, then the whole world of programming language academia\nis waiting for you. Languages and compilers have been studied formally since\nbefore we even had computers, so there is no shortage of books and papers on\nparser theory, type systems, semantics, and formal logic. Going down this\npath will also teach you how to read CS papers, which is a valuable skill in\nits own right.

    \n
    • \n
  • \n

    Or, if you just really enjoy hacking on and making languages, you can take\nLox and turn it into your own plaything. Change\nthe syntax to something that delights your eye. Add missing features or\nremove ones you don’t like. Jam new optimizations in there.

    \n\n

    Eventually you may get to a point where you have something you think others\ncould use as well. That gets you into the very distinct world of programming\nlanguage popularity. Expect to spend a ton of time writing documentation,\nexample programs, tools, and useful libraries. The field is crowded with\nlanguages vying for users. To thrive in that space you’ll have to put on\nyour marketing hat and sell. Not everyone enjoys that kind of\npublic-facing work, but if you do, it can be incredibly gratifying to see\npeople use your language to express themselves.

    \n
    • \n
\n

Or maybe this book has satisfied your craving and you’ll stop here. Whichever\nway you go, or don’t go, there is one lesson I hope to lodge in your heart. Like\nI was, you may have initially been intimidated by programming languages. But in\nthese chapters, you’ve seen that even really challenging material can be tackled\nby us mortals if we get our hands dirty and take it a step at a time. If you can\nhandle compilers and interpreters, you can do anything you put your mind to.

\n
\n

Challenges

\n

Assigning homework on the last day of school seems cruel but if you really want\nsomething to do during your summer vacation:

\n
    \n
  1. \n

    Fire up your profiler, run a couple of benchmarks, and look for other\nhotspots in the VM. Do you see anything in the runtime that you can improve?

    \n
    2. \n
  2. \n

    Many strings in real-world user programs are small, often only a character\nor two. This is less of a concern in clox because we intern strings, but\nmost VMs don’t. For those that don’t, heap allocating a tiny character array\nfor each of those little strings and then representing the value as a\npointer to that array is wasteful. Often, the pointer is larger than the\nstring’s characters. A classic trick is to have a separate value\nrepresentation for small strings that stores the characters inline in the\nvalue.

    \n

    Starting from clox’s original tagged union representation, implement that\noptimization. Write a couple of relevant benchmarks and see if it helps.

    \n
    3. \n
  3. \n

    Reflect back on your experience with this book. What parts of it worked well\nfor you? What didn’t? Was it easier for you to learn bottom-up or top-down?\nDid the illustrations help or distract? Did the analogies clarify or\nconfuse?

    \n

    The more you understand your personal learning style, the more effectively\nyou can upload knowledge into your head. You can specifically target\nmaterial that teaches you the way you learn best.

    \n
    4. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/parsing-expressions.html", "content": "\n\n\n\nParsing Expressions · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
6
\n

Parsing Expressions

\n\n
\n

Grammar, which knows how to control even kings.\nMolière

\n
\n

This chapter marks the first major milestone of the\nbook. Many of us have cobbled together a mishmash of regular expressions and\nsubstring operations to extract some sense out of a pile of text. The code was\nprobably riddled with bugs and a beast to maintain. Writing a real parser—one with decent error handling, a coherent internal structure, and the ability\nto robustly chew through a sophisticated syntax—is considered a rare,\nimpressive skill. In this chapter, you will attain\nit.

\n\n\n

It’s easier than you think, partially because we front-loaded a lot of the hard\nwork in the last chapter. You already know your way around a formal grammar.\nYou’re familiar with syntax trees, and we have some Java classes to represent\nthem. The only remaining piece is parsing—transmogrifying a sequence of\ntokens into one of those syntax trees.

\n

Some CS textbooks make a big deal out of parsers. In the ’60s, computer\nscientists—understandably tired of programming in assembly language—started designing more sophisticated, human-friendly\nlanguages like Fortran and ALGOL. Alas, they weren’t very machine-friendly\nfor the primitive computers of the time.

\n\n

These pioneers designed languages that they honestly weren’t even sure how to\nwrite compilers for, and then did groundbreaking work inventing parsing and\ncompiling techniques that could handle these new, big languages on those old, tiny\nmachines.

\n

Classic compiler books read like fawning hagiographies of these heroes and their\ntools. The cover of Compilers: Principles, Techniques, and Tools literally has\na dragon labeled “complexity of compiler design” being slain by a knight bearing\na sword and shield branded “LALR parser generator” and “syntax directed\ntranslation”. They laid it on thick.

\n

A little self-congratulation is well-deserved, but the truth is you don’t need\nto know most of that stuff to bang out a high quality parser for a modern\nmachine. As always, I encourage you to broaden your education and take it in\nlater, but this book omits the trophy case.

\n

6 . 1Ambiguity and the Parsing Game

\n

In the last chapter, I said you can “play” a context-free grammar like a game in\norder to generate strings. Parsers play that game in reverse. Given a string—a series of tokens—we map those tokens to terminals in the grammar to\nfigure out which rules could have generated that string.

\n

The “could have” part is interesting. It’s entirely possible to create a grammar\nthat is ambiguous, where different choices of productions can lead to the same\nstring. When you’re using the grammar to generate strings, that doesn’t matter\nmuch. Once you have the string, who cares how you got to it?

\n

When parsing, ambiguity means the parser may misunderstand the user’s code. As\nwe parse, we aren’t just determining if the string is valid Lox code, we’re\nalso tracking which rules match which parts of it so that we know what part of\nthe language each token belongs to. Here’s the Lox expression grammar we put\ntogether in the last chapter:

\n
expression     → literal\n               | unary\n               | binary\n               | grouping ;\n\nliteral        → NUMBER | STRING | "true" | "false" | "nil" ;\ngrouping       → "(" expression ")" ;\nunary          → ( "-" | "!" ) expression ;\nbinary         → expression operator expression ;\noperator       → "==" | "!=" | "<" | "<=" | ">" | ">="\n               | "+"  | "-"  | "*" | "/" ;\n
\n

This is a valid string in that grammar:

\"6\n

But there are two ways we could have generated it. One way is:

\n
    \n
  1. Starting at expression, pick binary.
    2. \n
  2. For the left-hand expression, pick NUMBER, and use 6.
    3. \n
  3. For the operator, pick \"/\".
    4. \n
  4. For the right-hand expression, pick binary again.
    5. \n
  5. In that nested binary expression, pick 3 - 1.
    6. \n
\n

Another is:

\n
    \n
  1. Starting at expression, pick binary.
    2. \n
  2. For the left-hand expression, pick binary again.
    3. \n
  3. In that nested binary expression, pick 6 / 3.
    4. \n
  4. Back at the outer binary, for the operator, pick \"-\".
    5. \n
  5. For the right-hand expression, pick NUMBER, and use 1.
    6. \n
\n

Those produce the same strings, but not the same syntax trees:

\"Two\n

In other words, the grammar allows seeing the expression as (6 / 3) - 1 or 6 / (3 - 1). The binary rule lets operands nest any which way you want. That in\nturn affects the result of evaluating the parsed tree. The way mathematicians\nhave addressed this ambiguity since blackboards were first invented is by\ndefining rules for precedence and associativity.

\n
    \n
  • \n

    Precedence determines which operator\nis evaluated first in an expression containing a mixture of different\noperators. Precedence rules tell us that we evaluate the / before the -\nin the above example. Operators with higher precedence are evaluated\nbefore operators with lower precedence. Equivalently, higher precedence\noperators are said to “bind tighter”.

    \n
    • \n
  • \n

    Associativity determines which operator is evaluated first in a series\nof the same operator. When an operator is left-associative (think\n“left-to-right”), operators on the left evaluate before those on the right.\nSince - is left-associative, this expression:

    \n
    5 - 3 - 1\n
    \n

    is equivalent to:

    \n
    (5 - 3) - 1\n
    \n

    Assignment, on the other hand, is right-associative. This:

    \n
    a = b = c\n
    \n

    is equivalent to:

    \n
    a = (b = c)\n
    \n
    • \n
\n\n

Without well-defined precedence and associativity, an expression that uses\nmultiple operators is ambiguous—it can be parsed into different syntax trees,\nwhich could in turn evaluate to different results. We’ll fix that in Lox by\napplying the same precedence rules as C, going from lowest to highest.

\n\n\n \n \n \n\n\n\n\n \n \n \n\n\n \n \n \n\n\n \n \n \n\n\n \n \n \n\n\n \n \n \n\n\n
NameOperatorsAssociates
Equality== !=Left
Comparison> >=\n < <=Left
Term- +Left
Factor/ *Left
Unary! -Right
\n

Right now, the grammar stuffs all expression types into a single expression\nrule. That same rule is used as the non-terminal for operands, which lets the\ngrammar accept any kind of expression as a subexpression, regardless of whether\nthe precedence rules allow it.

\n

We fix that by stratifying the grammar. We define a\nseparate rule for each precedence level.

\n
expression     → ...\nequality       → ...\ncomparison     → ...\nterm           → ...\nfactor         → ...\nunary          → ...\nprimary        → ...\n
\n\n

Each rule here only matches expressions at its precedence level or higher. For\nexample, unary matches a unary expression like !negated or a primary\nexpression like 1234. And term can match 1 + 2 but also 3 * 4 / 5. The\nfinal primary rule covers the highest-precedence forms—literals and\nparenthesized expressions.

\n

We just need to fill in the productions for each of those rules. We’ll do the\neasy ones first. The top expression rule matches any expression at any\nprecedence level. Since equality has the lowest\nprecedence, if we match that, then it covers everything.

\n\n
expression     → equality\n
\n

Over at the other end of the precedence table, a primary expression contains\nall the literals and grouping expressions.

\n
primary        → NUMBER | STRING | "true" | "false" | "nil"\n               | "(" expression ")" ;\n
\n

A unary expression starts with a unary operator followed by the operand. Since\nunary operators can nest—!!true is a valid if weird expression—the\noperand can itself be a unary operator. A recursive rule handles that nicely.

\n
unary          → ( "!" | "-" ) unary ;\n
\n

But this rule has a problem. It never terminates.

\n

Remember, each rule needs to match expressions at that precedence level or\nhigher, so we also need to let this match a primary expression.

\n
unary          → ( "!" | "-" ) unary\n               | primary ;\n
\n

That works.

\n

The remaining rules are all binary operators. We’ll start with the rule for\nmultiplication and division. Here’s a first try:

\n
factor         → factor ( "/" | "*" ) unary\n               | unary ;\n
\n

The rule recurses to match the left operand. That enables the rule to match a\nseries of multiplication and division expressions like 1 * 2 / 3. Putting the\nrecursive production on the left side and unary on the right makes the rule\nleft-associative and unambiguous.

\n\n

All of this is correct, but the fact that the first symbol in the body of the\nrule is the same as the head of the rule means this production is\nleft-recursive. Some parsing techniques, including the one we’re going to\nuse, have trouble with left recursion. (Recursion elsewhere, like we have in\nunary and the indirect recursion for grouping in primary are not a problem.)

\n

There are many grammars you can define that match the same language. The choice\nfor how to model a particular language is partially a matter of taste and\npartially a pragmatic one. This rule is correct, but not optimal for how we\nintend to parse it. Instead of a left recursive rule, we’ll use a different one.

\n
factor         → unary ( ( "/" | "*" ) unary )* ;\n
\n

We define a factor expression as a flat sequence of multiplications\nand divisions. This matches the same syntax as the previous rule, but better\nmirrors the code we’ll write to parse Lox. We use the same structure for all of\nthe other binary operator precedence levels, giving us this complete expression\ngrammar:

\n
expression     → equality ;\nequality       → comparison ( ( "!=" | "==" ) comparison )* ;\ncomparison     → term ( ( ">" | ">=" | "<" | "<=" ) term )* ;\nterm           → factor ( ( "-" | "+" ) factor )* ;\nfactor         → unary ( ( "/" | "*" ) unary )* ;\nunary          → ( "!" | "-" ) unary\n               | primary ;\nprimary        → NUMBER | STRING | "true" | "false" | "nil"\n               | "(" expression ")" ;\n
\n

This grammar is more complex than the one we had before, but in return we have\neliminated the previous one’s ambiguity. It’s just what we need to make a\nparser.

\n

6 . 2Recursive Descent Parsing

\n

There is a whole pack of parsing techniques whose names are mostly combinations\nof “L” and “R”—LL(k), LR(1), LALR—along with more exotic\nbeasts like parser combinators, Earley parsers, the shunting yard\nalgorithm, and packrat parsing. For our first interpreter, one\ntechnique is more than sufficient: recursive descent.

\n

Recursive descent is the simplest way to build a parser, and doesn’t require\nusing complex parser generator tools like Yacc, Bison or ANTLR. All you need is\nstraightforward handwritten code. Don’t be fooled by its simplicity, though.\nRecursive descent parsers are fast, robust, and can support sophisticated\nerror handling. In fact, GCC, V8 (the JavaScript VM in Chrome), Roslyn (the C#\ncompiler written in C#) and many other heavyweight production language\nimplementations use recursive descent. It rocks.

\n

Recursive descent is considered a top-down parser because it starts from the\ntop or outermost grammar rule (here expression) and works its way down into the nested subexpressions before finally\nreaching the leaves of the syntax tree. This is in contrast with bottom-up\nparsers like LR that start with primary expressions and compose them into larger\nand larger chunks of syntax.

\n\n

A recursive descent parser is a literal translation of the grammar’s rules\nstraight into imperative code. Each rule becomes a function. The body of the\nrule translates to code roughly like:

\n\n\n \n \n\n\n\n \n \n \n \n \n\n
Grammar notationCode representation
TerminalCode to match and consume a token
NonterminalCall to that rule’s function
|if or switch statement
* or +while or for loop
?if statement
\n

The descent is described as “recursive” because when a grammar rule refers to\nitself—directly or indirectly—that translates to a recursive function\ncall.

\n

6 . 2 . 1The parser class

\n

Each grammar rule becomes a method inside this new class:

\n
lox/Parser.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.util.List;\n\nimport static com.craftinginterpreters.lox.TokenType.*;\n\nclass Parser {\n  private final List<Token> tokens;\n  private int current = 0;\n\n  Parser(List<Token> tokens) {\n    this.tokens = tokens;\n  }\n}\n
\n
lox/Parser.java, create new file
\n\n

Like the scanner, the parser consumes a flat input sequence, only now we’re\nreading tokens instead of characters. We store the list of tokens and use\ncurrent to point to the next token eagerly waiting to be parsed.

\n

We’re going to run straight through the expression grammar now and translate\neach rule to Java code. The first rule, expression, simply expands to the\nequality rule, so that’s straightforward.

\n
lox/Parser.java
\nadd after Parser()
\n
  private Expr expression() {\n    return equality();\n  }\n
\n
lox/Parser.java, add after Parser()
\n\n

Each method for parsing a grammar rule produces a syntax tree for that rule and\nreturns it to the caller. When the body of the rule contains a nonterminal—a\nreference to another rule—we call that other rule’s\nmethod.

\n\n

The rule for equality is a little more complex.

\n
equality       → comparison ( ( "!=" | "==" ) comparison )* ;\n
\n

In Java, that becomes:

\n
lox/Parser.java
\nadd after expression()
\n
  private Expr equality() {\n    Expr expr = comparison();\n\n    while (match(BANG_EQUAL, EQUAL_EQUAL)) {\n      Token operator = previous();\n      Expr right = comparison();\n      expr = new Expr.Binary(expr, operator, right);\n    }\n\n    return expr;\n  }\n
\n
lox/Parser.java, add after expression()
\n\n

Let’s step through it. The first comparison nonterminal in the body translates\nto the first call to comparison() in the method. We take that result and store\nit in a local variable.

\n

Then, the ( ... )* loop in the rule maps to a while loop. We need to know\nwhen to exit that loop. We can see that inside the rule, we must first find\neither a != or == token. So, if we don’t see one of those, we must be done\nwith the sequence of equality operators. We express that check using a handy\nmatch() method.

\n
lox/Parser.java
\nadd after equality()
\n
  private boolean match(TokenType... types) {\n    for (TokenType type : types) {\n      if (check(type)) {\n        advance();\n        return true;\n      }\n    }\n\n    return false;\n  }\n
\n
lox/Parser.java, add after equality()
\n\n

This checks to see if the current token has any of the given types. If so, it\nconsumes the token and returns true. Otherwise, it returns false and leaves\nthe current token alone. The match() method is defined in terms of two more\nfundamental operations.

\n

The check() method returns true if the current token is of the given type.\nUnlike match(), it never consumes the token, it only looks at it.

\n
lox/Parser.java
\nadd after match()
\n
  private boolean check(TokenType type) {\n    if (isAtEnd()) return false;\n    return peek().type == type;\n  }\n
\n
lox/Parser.java, add after match()
\n\n

The advance() method consumes the current token and returns it, similar to how\nour scanner’s corresponding method crawled through characters.

\n
lox/Parser.java
\nadd after check()
\n
  private Token advance() {\n    if (!isAtEnd()) current++;\n    return previous();\n  }\n
\n
lox/Parser.java, add after check()
\n\n

These methods bottom out on the last handful of primitive operations.

\n
lox/Parser.java
\nadd after advance()
\n
  private boolean isAtEnd() {\n    return peek().type == EOF;\n  }\n\n  private Token peek() {\n    return tokens.get(current);\n  }\n\n  private Token previous() {\n    return tokens.get(current - 1);\n  }\n
\n
lox/Parser.java, add after advance()
\n\n

isAtEnd() checks if we’ve run out of tokens to parse. peek() returns the\ncurrent token we have yet to consume, and previous() returns the most recently\nconsumed token. The latter makes it easier to use match() and then access the\njust-matched token.

\n

That’s most of the parsing infrastructure we need. Where were we? Right, so if\nwe are inside the while loop in equality(), then we know we have found a\n!= or == operator and must be parsing an equality expression.

\n

We grab the matched operator token so we can track which kind of equality\nexpression we have. Then we call comparison() again to parse the right-hand\noperand. We combine the operator and its two operands into a new Expr.Binary\nsyntax tree node, and then loop around. For each iteration, we store the\nresulting expression back in the same expr local variable. As we zip through a\nsequence of equality expressions, that creates a left-associative nested tree of\nbinary operator nodes.

\n

\"The\n\n

The parser falls out of the loop once it hits a token that’s not an equality\noperator. Finally, it returns the expression. Note that if the parser never\nencounters an equality operator, then it never enters the loop. In that case,\nthe equality() method effectively calls and returns comparison(). In that\nway, this method matches an equality operator or anything of higher\nprecedence.

\n

Moving on to the next rule . . . 

\n
comparison     → term ( ( ">" | ">=" | "<" | "<=" ) term )* ;\n
\n

Translated to Java:

\n
lox/Parser.java
\nadd after equality()
\n
  private Expr comparison() {\n    Expr expr = term();\n\n    while (match(GREATER, GREATER_EQUAL, LESS, LESS_EQUAL)) {\n      Token operator = previous();\n      Expr right = term();\n      expr = new Expr.Binary(expr, operator, right);\n    }\n\n    return expr;\n  }\n
\n
lox/Parser.java, add after equality()
\n\n

The grammar rule is virtually identical to equality\nand so is the corresponding code. The only differences are the token types for\nthe operators we match, and the method we call for the operands—now\nterm() instead of comparison(). The remaining two binary operator rules\nfollow the same pattern.

\n

In order of precedence, first addition and subtraction:

\n\n
lox/Parser.java
\nadd after comparison()
\n
  private Expr term() {\n    Expr expr = factor();\n\n    while (match(MINUS, PLUS)) {\n      Token operator = previous();\n      Expr right = factor();\n      expr = new Expr.Binary(expr, operator, right);\n    }\n\n    return expr;\n  }\n
\n
lox/Parser.java, add after comparison()
\n\n

And finally, multiplication and division:

\n
lox/Parser.java
\nadd after term()
\n
  private Expr factor() {\n    Expr expr = unary();\n\n    while (match(SLASH, STAR)) {\n      Token operator = previous();\n      Expr right = unary();\n      expr = new Expr.Binary(expr, operator, right);\n    }\n\n    return expr;\n  }\n
\n
lox/Parser.java, add after term()
\n\n

That’s all of the binary operators, parsed with the correct precedence and\nassociativity. We’re crawling up the precedence hierarchy and now we’ve reached\nthe unary operators.

\n
unary          → ( "!" | "-" ) unary\n               | primary ;\n
\n

The code for this is a little different.

\n
lox/Parser.java
\nadd after factor()
\n
  private Expr unary() {\n    if (match(BANG, MINUS)) {\n      Token operator = previous();\n      Expr right = unary();\n      return new Expr.Unary(operator, right);\n    }\n\n    return primary();\n  }\n
\n
lox/Parser.java, add after factor()
\n\n

Again, we look at the current token to see how to\nparse. If it’s a ! or -, we must have a unary expression. In that case, we\ngrab the token and then recursively call unary() again to parse the operand.\nWrap that all up in a unary expression syntax tree and we’re done.

\n\n

Otherwise, we must have reached the highest level of precedence, primary\nexpressions.

\n
primary        → NUMBER | STRING | "true" | "false" | "nil"\n               | "(" expression ")" ;\n
\n

Most of the cases for the rule are single terminals, so parsing is\nstraightforward.

\n
lox/Parser.java
\nadd after unary()
\n
  private Expr primary() {\n    if (match(FALSE)) return new Expr.Literal(false);\n    if (match(TRUE)) return new Expr.Literal(true);\n    if (match(NIL)) return new Expr.Literal(null);\n\n    if (match(NUMBER, STRING)) {\n      return new Expr.Literal(previous().literal);\n    }\n\n    if (match(LEFT_PAREN)) {\n      Expr expr = expression();\n      consume(RIGHT_PAREN, "Expect ')' after expression.");\n      return new Expr.Grouping(expr);\n    }\n  }\n
\n
lox/Parser.java, add after unary()
\n\n

The interesting branch is the one for handling parentheses. After we match an\nopening ( and parse the expression inside it, we must find a ) token. If\nwe don’t, that’s an error.

\n

6 . 3Syntax Errors

\n

A parser really has two jobs:

\n
    \n
  1. \n

    Given a valid sequence of tokens, produce a corresponding syntax tree.

    \n
    2. \n
  2. \n

    Given an invalid sequence of tokens, detect any errors and tell the\nuser about their mistakes.

    \n
    3. \n
\n

Don’t underestimate how important the second job is! In modern IDEs and editors,\nthe parser is constantly reparsing code—often while the user is still editing\nit—in order to syntax highlight and support things like auto-complete. That\nmeans it will encounter code in incomplete, half-wrong states all the time.

\n

When the user doesn’t realize the syntax is wrong, it is up to the parser to\nhelp guide them back onto the right path. The way it reports errors is a large\npart of your language’s user interface. Good syntax error handling is hard. By\ndefinition, the code isn’t in a well-defined state, so there’s no infallible way\nto know what the user meant to write. The parser can’t read your mind.

\n\n

There are a couple of hard requirements for when the parser runs into a syntax\nerror. A parser must:

\n
    \n
  • \n

    Detect and report the error. If it doesn’t detect the error and passes the resulting malformed syntax tree on\nto the interpreter, all manner of horrors may be summoned.

    \n
    • \n
  • \n

    Avoid crashing or hanging. Syntax errors are a fact of life, and\nlanguage tools have to be robust in the face of them. Segfaulting or getting\nstuck in an infinite loop isn’t allowed. While the source may not be valid\ncode, it’s still a valid input to the parser because users use the\nparser to learn what syntax is allowed.

    \n
    • \n
\n

Those are the table stakes if you want to get in the parser game at all, but you\nreally want to raise the ante beyond that. A decent parser should:

\n
    \n
  • \n

    Be fast. Computers are thousands of times faster than they were when\nparser technology was first invented. The days of needing to optimize your\nparser so that it could get through an entire source file during a coffee\nbreak are over. But programmer expectations have risen as quickly, if not\nfaster. They expect their editors to reparse files in milliseconds after\nevery keystroke.

    \n
    • \n
  • \n

    Report as many distinct errors as there are. Aborting after the first\nerror is easy to implement, but it’s annoying for users if every time they\nfix what they think is the one error in a file, a new one appears. They\nwant to see them all.

    \n
    • \n
  • \n

    Minimize cascaded errors. Once a single error is found, the parser no\nlonger really knows what’s going on. It tries to get itself back on track\nand keep going, but if it gets confused, it may report a slew of ghost\nerrors that don’t indicate other real problems in the code. When the first\nerror is fixed, those phantoms disappear, because they reflect only the\nparser’s own confusion. Cascaded errors are annoying because they can scare\nthe user into thinking their code is in a worse state than it is.

    \n
    • \n
\n

The last two points are in tension. We want to report as many separate errors as\nwe can, but we don’t want to report ones that are merely side effects of an\nearlier one.

\n

The way a parser responds to an error and keeps going to look for later errors\nis called error recovery. This was a hot research topic in the ’60s. Back\nthen, you’d hand a stack of punch cards to the secretary and come back the next\nday to see if the compiler succeeded. With an iteration loop that slow, you\nreally wanted to find every single error in your code in one pass.

\n

Today, when parsers complete before you’ve even finished typing, it’s less of an\nissue. Simple, fast error recovery is fine.

\n

6 . 3 . 1Panic mode error recovery

\n\n

Of all the recovery techniques devised in yesteryear, the one that best stood\nthe test of time is called—somewhat alarmingly—panic\nmode. As soon as the parser detects an error, it enters panic mode. It\nknows at least one token doesn’t make sense given its current state in the\nmiddle of some stack of grammar productions.

\n

Before it can get back to parsing, it needs to get its state and the sequence of\nforthcoming tokens aligned such that the next token does match the rule being\nparsed. This process is called synchronization.

\n

To do that, we select some rule in the grammar that will mark the\nsynchronization point. The parser fixes its parsing state by jumping out of any\nnested productions until it gets back to that rule. Then it synchronizes the\ntoken stream by discarding tokens until it reaches one that can appear at that\npoint in the rule.

\n

Any additional real syntax errors hiding in those discarded tokens aren’t\nreported, but it also means that any mistaken cascaded errors that are side\neffects of the initial error aren’t falsely reported either, which is a decent\ntrade-off.

\n

The traditional place in the grammar to synchronize is between statements. We\ndon’t have those yet, so we won’t actually synchronize in this chapter, but\nwe’ll get the machinery in place for later.

\n

6 . 3 . 2Entering panic mode

\n

Back before we went on this side trip around error recovery, we were writing the\ncode to parse a parenthesized expression. After parsing the expression, the\nparser looks for the closing ) by calling consume(). Here, finally, is that\nmethod:

\n
lox/Parser.java
\nadd after match()
\n
  private Token consume(TokenType type, String message) {\n    if (check(type)) return advance();\n\n    throw error(peek(), message);\n  }\n
\n
lox/Parser.java, add after match()
\n\n

It’s similar to match() in that it checks to see if the next token is of the\nexpected type. If so, it consumes the token and everything is groovy. If some\nother token is there, then we’ve hit an error. We report it by calling this:

\n
lox/Parser.java
\nadd after previous()
\n
  private ParseError error(Token token, String message) {\n    Lox.error(token, message);\n    return new ParseError();\n  }\n
\n
lox/Parser.java, add after previous()
\n\n

First, that shows the error to the user by calling:

\n
lox/Lox.java
\nadd after report()
\n
  static void error(Token token, String message) {\n    if (token.type == TokenType.EOF) {\n      report(token.line, " at end", message);\n    } else {\n      report(token.line, " at '" + token.lexeme + "'", message);\n    }\n  }\n
\n
lox/Lox.java, add after report()
\n\n

This reports an error at a given token. It shows the token’s location and the\ntoken itself. This will come in handy later since we use tokens throughout the\ninterpreter to track locations in code.

\n

After we report the error, the user knows about their mistake, but what does the\nparser do next? Back in error(), we create and return a ParseError, an\ninstance of this new class:

\n
class Parser {\n
lox/Parser.java
\nnest inside class Parser
\n
  private static class ParseError extends RuntimeException {}\n\n
  private final List<Token> tokens;\n
\n
lox/Parser.java, nest inside class Parser
\n\n

This is a simple sentinel class we use to unwind the parser. The error()\nmethod returns the error instead of throwing it because we want to let the\ncalling method inside the parser decide whether to unwind or not. Some parse\nerrors occur in places where the parser isn’t likely to get into a weird state\nand we don’t need to synchronize. In those\nplaces, we simply report the error and keep on truckin’.

\n

For example, Lox limits the number of arguments you can pass to a function. If\nyou pass too many, the parser needs to report that error, but it can and should\nsimply keep on parsing the extra arguments instead of freaking out and going\ninto panic mode.

\n\n

In our case, though, the syntax error is nasty enough that we want to panic and\nsynchronize. Discarding tokens is pretty easy, but how do we synchronize the\nparser’s own state?

\n

6 . 3 . 3Synchronizing a recursive descent parser

\n

With recursive descent, the parser’s state—which rules it is in the middle of\nrecognizing—is not stored explicitly in fields. Instead, we use Java’s\nown call stack to track what the parser is doing. Each rule in the middle of\nbeing parsed is a call frame on the stack. In order to reset that state, we need\nto clear out those call frames.

\n

The natural way to do that in Java is exceptions. When we want to synchronize,\nwe throw that ParseError object. Higher up in the method for the grammar rule\nwe are synchronizing to, we’ll catch it. Since we synchronize on statement\nboundaries, we’ll catch the exception there. After the exception is caught, the\nparser is in the right state. All that’s left is to synchronize the tokens.

\n

We want to discard tokens until we’re right at the beginning of the next\nstatement. That boundary is pretty easy to spot—it’s one of the main reasons\nwe picked it. After a semicolon, we’re probably\nfinished with a statement. Most statements start with a keyword—for, if,\nreturn, var, etc. When the next token is any of those, we’re probably\nabout to start a statement.

\n\n

This method encapsulates that logic:

\n
lox/Parser.java
\nadd after error()
\n
  private void synchronize() {\n    advance();\n\n    while (!isAtEnd()) {\n      if (previous().type == SEMICOLON) return;\n\n      switch (peek().type) {\n        case CLASS:\n        case FUN:\n        case VAR:\n        case FOR:\n        case IF:\n        case WHILE:\n        case PRINT:\n        case RETURN:\n          return;\n      }\n\n      advance();\n    }\n  }\n
\n
lox/Parser.java, add after error()
\n\n

It discards tokens until it thinks it has found a statement boundary. After\ncatching a ParseError, we’ll call this and then we are hopefully back in sync.\nWhen it works well, we have discarded tokens that would have likely caused\ncascaded errors anyway, and now we can parse the rest of the file starting at\nthe next statement.

\n

Alas, we don’t get to see this method in action, since we don’t have statements\nyet. We’ll get to that in a couple of chapters. For now, if an\nerror occurs, we’ll panic and unwind all the way to the top and stop parsing.\nSince we can parse only a single expression anyway, that’s no big loss.

\n

6 . 4Wiring up the Parser

\n

We are mostly done parsing expressions now. There is one other place where we\nneed to add a little error handling. As the parser descends through the parsing\nmethods for each grammar rule, it eventually hits primary(). If none of the\ncases in there match, it means we are sitting on a token that can’t start an\nexpression. We need to handle that error too.

\n
    if (match(LEFT_PAREN)) {\n      Expr expr = expression();\n      consume(RIGHT_PAREN, "Expect ')' after expression.");\n      return new Expr.Grouping(expr);\n    }\n
lox/Parser.java
\nin primary()
\n
\n\n    throw error(peek(), "Expect expression.");\n
  }\n
\n
lox/Parser.java, in primary()
\n\n

With that, all that remains in the parser is to define an initial method to kick\nit off. That method is called, naturally enough, parse().

\n
lox/Parser.java
\nadd after Parser()
\n
  Expr parse() {\n    try {\n      return expression();\n    } catch (ParseError error) {\n      return null;\n    }\n  }\n
\n
lox/Parser.java, add after Parser()
\n\n

We’ll revisit this method later when we add statements to the language. For now,\nit parses a single expression and returns it. We also have some temporary code\nto exit out of panic mode. Syntax error recovery is the parser’s job, so we\ndon’t want the ParseError exception to escape into the rest of the interpreter.

\n

When a syntax error does occur, this method returns null. That’s OK. The\nparser promises not to crash or hang on invalid syntax, but it doesn’t promise\nto return a usable syntax tree if an error is found. As soon as the parser\nreports an error, hadError gets set, and subsequent phases are skipped.

\n

Finally, we can hook up our brand new parser to the main Lox class and try it\nout. We still don’t have an interpreter, so for now, we’ll parse to a syntax\ntree and then use the AstPrinter class from the last chapter to\ndisplay it.

\n

Delete the old code to print the scanned tokens and replace it with this:

\n
    List<Token> tokens = scanner.scanTokens();\n
lox/Lox.java
\nin run()
\nreplace 5 lines
\n
    Parser parser = new Parser(tokens);\n    Expr expression = parser.parse();\n\n    // Stop if there was a syntax error.\n    if (hadError) return;\n\n    System.out.println(new AstPrinter().print(expression));\n
  }\n
\n
lox/Lox.java, in run(), replace 5 lines
\n\n

Congratulations, you have crossed the threshold! That\nreally is all there is to handwriting a parser. We’ll extend the grammar in\nlater chapters with assignment, statements, and other stuff, but none of that is\nany more complex than the binary operators we tackled here.

\n\n

Fire up the interpreter and type in some expressions. See how it handles\nprecedence and associativity correctly? Not bad for less than 200 lines of code.

\n
\n

Challenges

\n
    \n
  1. \n

    In C, a block is a statement form that allows you to pack a series of\nstatements where a single one is expected. The comma operator is an\nanalogous syntax for expressions. A comma-separated series of expressions\ncan be given where a single expression is expected (except inside a function\ncall’s argument list). At runtime, the comma operator evaluates the left\noperand and discards the result. Then it evaluates and returns the right\noperand.

    \n

    Add support for comma expressions. Give them the same precedence and\nassociativity as in C. Write the grammar, and then implement the necessary\nparsing code.

    \n
    2. \n
  2. \n

    Likewise, add support for the C-style conditional or “ternary” operator\n?:. What precedence level is allowed between the ? and :? Is the whole\noperator left-associative or right-associative?

    \n
    3. \n
  3. \n

    Add error productions to handle each binary operator appearing without a\nleft-hand operand. In other words, detect a binary operator appearing at the\nbeginning of an expression. Report that as an error, but also parse and\ndiscard a right-hand operand with the appropriate precedence.

    \n
    4. \n
\n
\n
\n

Design Note: Logic Versus History

\n

Let’s say we decide to add bitwise & and | operators to Lox. Where should we\nput them in the precedence hierarchy? C—and most languages that follow in C’s\nfootsteps—place them below ==. This is widely considered a mistake because\nit means common operations like testing a flag require parentheses.

\n
if (flags & FLAG_MASK == SOME_FLAG) { ... } // Wrong.\nif ((flags & FLAG_MASK) == SOME_FLAG) { ... } // Right.\n
\n

Should we fix this for Lox and put bitwise operators higher up the precedence\ntable than C does? There are two strategies we can take.

\n

You almost never want to use the result of an == expression as the operand to\na bitwise operator. By making bitwise bind tighter, users don’t need to\nparenthesize as often. So if we do that, and users assume the precedence is\nchosen logically to minimize parentheses, they’re likely to infer it correctly.

\n

This kind of internal consistency makes the language easier to learn because\nthere are fewer edge cases and exceptions users have to stumble into and then\ncorrect. That’s good, because before users can use our language, they have to\nload all of that syntax and semantics into their heads. A simpler, more rational\nlanguage makes sense.

\n

But, for many users there is an even faster shortcut to getting our language’s\nideas into their wetware—use concepts they already know. Many newcomers to\nour language will be coming from some other language or languages. If our\nlanguage uses some of the same syntax or semantics as those, there is much less\nfor the user to learn (and unlearn).

\n

This is particularly helpful with syntax. You may not remember it well today,\nbut way back when you learned your very first programming language, code\nprobably looked alien and unapproachable. Only through painstaking effort did\nyou learn to read and accept it. If you design a novel syntax for your new\nlanguage, you force users to start that process all over again.

\n

Taking advantage of what users already know is one of the most powerful tools\nyou can use to ease adoption of your language. It’s almost impossible to\noverestimate how valuable this is. But it faces you with a nasty problem: What\nhappens when the thing the users all know kind of sucks? C’s bitwise operator\nprecedence is a mistake that doesn’t make sense. But it’s a familiar mistake\nthat millions have already gotten used to and learned to live with.

\n

Do you stay true to your language’s own internal logic and ignore history? Do\nyou start from a blank slate and first principles? Or do you weave your language\ninto the rich tapestry of programming history and give your users a leg up by\nstarting from something they already know?

\n

There is no perfect answer here, only trade-offs. You and I are obviously biased\ntowards liking novel languages, so our natural inclination is to burn the\nhistory books and start our own story.

\n

In practice, it’s often better to make the most of what users already know.\nGetting them to come to your language requires a big leap. The smaller you can\nmake that chasm, the more people will be willing to cross it. But you can’t\nalways stick to history, or your language won’t have anything new and\ncompelling to give people a reason to jump over.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/representing-code.html", "content": "\n\n\n\nRepresenting Code · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
5
\n

Representing Code

\n\n
\n

To dwellers in a wood, almost every species of tree has its voice as well as\nits feature.\nThomas Hardy, Under the Greenwood Tree

\n
\n

In the last chapter, we took the raw source code as a string and\ntransformed it into a slightly higher-level representation: a series of tokens.\nThe parser we’ll write in the next chapter takes those tokens and\ntransforms them yet again, into an even richer, more complex representation.

\n

Before we can produce that representation, we need to define it. That’s the\nsubject of this chapter. Along the way, we’ll cover\nsome theory around formal grammars, feel the difference between functional and\nobject-oriented programming, go over a couple of design patterns, and do some\nmetaprogramming.

\n\n

Before we do all that, let’s focus on the main goal—a representation for\ncode. It should be simple for the parser to produce and easy for the\ninterpreter to consume. If you haven’t written a parser or interpreter yet,\nthose requirements aren’t exactly illuminating. Maybe your intuition can help.\nWhat is your brain doing when you play the part of a human interpreter? How do\nyou mentally evaluate an arithmetic expression like this:

\n
1 + 2 * 3 - 4\n
\n

Because you understand the order of operations—the old “Please Excuse My\nDear Aunt Sally” stuff—you know that the multiplication is evaluated\nbefore the addition or subtraction. One way to visualize that precedence is\nusing a tree. Leaf nodes are numbers, and interior nodes are operators with\nbranches for each of their operands.

\n

In order to evaluate an arithmetic node, you need to know the numeric values of\nits subtrees, so you have to evaluate those first. That means working your way\nfrom the leaves up to the root—a post-order traversal:

\n

\"Evaluating\n\n

If I gave you an arithmetic expression, you could draw one of these trees pretty\neasily. Given a tree, you can evaluate it without breaking a sweat. So it\nintuitively seems like a workable representation of our code is a tree that matches the grammatical structure—the operator\nnesting—of the language.

\n\n

We need to get more precise about what that grammar is then. Like lexical\ngrammars in the last chapter, there is a long ton of theory around syntactic\ngrammars. We’re going into that theory a little more than we did when scanning\nbecause it turns out to be a useful tool throughout much of the interpreter.\nWe start by moving one level up the Chomsky hierarchy . . . 

\n

5 . 1Context-Free Grammars

\n

In the last chapter, the formalism we used for defining the lexical grammar—the rules for how characters get grouped into tokens—was called a regular\nlanguage. That was fine for our scanner, which emits a flat sequence of tokens.\nBut regular languages aren’t powerful enough to handle expressions which can\nnest arbitrarily deeply.

\n

We need a bigger hammer, and that hammer is a context-free grammar\n(CFG). It’s the next heaviest tool in the toolbox of\nformal grammars. A formal grammar takes a set of atomic pieces it calls\nits “alphabet”. Then it defines a (usually infinite) set of “strings” that are\n“in” the grammar. Each string is a sequence of “letters” in the alphabet.

\n

I’m using all those quotes because the terms get a little confusing as you move\nfrom lexical to syntactic grammars. In our scanner’s grammar, the alphabet\nconsists of individual characters and the strings are the valid lexemes—roughly “words”. In the syntactic grammar we’re talking about now, we’re at a\ndifferent level of granularity. Now each “letter” in the alphabet is an entire\ntoken and a “string” is a sequence of tokens—an entire expression.

\n

Oof. Maybe a table will help:

\n\n\n \n \n \n \n\n\n\n\n \n \n \n \n\n\n \n \n \n \n\n\n \n \n \n \n\n\n
TerminologyLexical grammarSyntactic grammar
The “alphabet” is . . .→ CharactersTokens
A “string” is . . .→ Lexeme or tokenExpression
It’s implemented by the . . .→ ScannerParser
\n

A formal grammar’s job is to specify which strings are valid and which aren’t.\nIf we were defining a grammar for English sentences, “eggs are tasty for\nbreakfast” would be in the grammar, but “tasty breakfast for are eggs” would\nprobably not.

\n

5 . 1 . 1Rules for grammars

\n

How do we write down a grammar that contains an infinite number of valid\nstrings? We obviously can’t list them all out. Instead, we create a finite set\nof rules. You can think of them as a game that you can “play” in one of two\ndirections.

\n

If you start with the rules, you can use them to generate strings that are in\nthe grammar. Strings created this way are called derivations because each is\nderived from the rules of the grammar. In each step of the game, you pick a\nrule and follow what it tells you to do. Most of the lingo around formal\ngrammars comes from playing them in this direction. Rules are called\nproductions because they produce strings in the grammar.

\n

Each production in a context-free grammar has a head—its name—and a body, which describes what it generates. In\nits pure form, the body is simply a list of symbols. Symbols come in two\ndelectable flavors:

\n\n
    \n
  • \n

    A terminal is a letter from the grammar’s alphabet. You can think of it\nlike a literal value. In the syntactic grammar we’re defining, the terminals\nare individual lexemes—tokens coming from the scanner like if or\n1234.

    \n

    These are called “terminals”, in the sense of an “end point” because they\ndon’t lead to any further “moves” in the game. You simply produce that one\nsymbol.

    \n
    • \n
  • \n

    A nonterminal is a named reference to another rule in the grammar. It\nmeans “play that rule and insert whatever it produces here”. In this way,\nthe grammar composes.

    \n
    • \n
\n

There is one last refinement: you may have multiple rules with the same name.\nWhen you reach a nonterminal with that name, you are allowed to pick any of the\nrules for it, whichever floats your boat.

\n

To make this concrete, we need a way to write down\nthese production rules. People have been trying to crystallize grammar all the\nway back to Pāṇini’s Ashtadhyayi, which codified Sanskrit grammar a mere\ncouple thousand years ago. Not much progress happened until John Backus and\ncompany needed a notation for specifying ALGOL 58 and came up with\nBackus-Naur form (BNF). Since then, nearly everyone uses some\nflavor of BNF, tweaked to their own tastes.

\n

I tried to come up with something clean. Each rule is a name, followed by an\narrow (→), followed by a sequence of symbols, and finally ending with a\nsemicolon (;). Terminals are quoted strings, and nonterminals are lowercase\nwords.

\n\n

Using that, here’s a grammar for breakfast menus:

\n\n
breakfast  → protein "with" breakfast "on the side" ;\nbreakfast  → protein ;\nbreakfast  → bread ;\n\nprotein    → crispiness "crispy" "bacon" ;\nprotein    → "sausage" ;\nprotein    → cooked "eggs" ;\n\ncrispiness → "really" ;\ncrispiness → "really" crispiness ;\n\ncooked     → "scrambled" ;\ncooked     → "poached" ;\ncooked     → "fried" ;\n\nbread      → "toast" ;\nbread      → "biscuits" ;\nbread      → "English muffin" ;\n
\n

We can use this grammar to generate random breakfasts. Let’s play a round and\nsee how it works. By age-old convention, the game starts with the first rule in\nthe grammar, here breakfast. There are three productions for that, and we\nrandomly pick the first one. Our resulting string looks like:

\n
protein "with" breakfast "on the side"\n
\n

We need to expand that first nonterminal, protein, so we pick a production for\nthat. Let’s pick:

\n
protein → cooked "eggs" ;\n
\n

Next, we need a production for cooked, and so we pick \"poached\". That’s a\nterminal, so we add that. Now our string looks like:

\n
"poached" "eggs" "with" breakfast "on the side"\n
\n

The next non-terminal is breakfast again. The first breakfast production we\nchose recursively refers back to the breakfast rule. Recursion in the grammar\nis a good sign that the language being defined is context-free instead of\nregular. In particular, recursion where the recursive nonterminal has\nproductions on both sides implies that the language is\nnot regular.

\n\n

We could keep picking the first production for breakfast over and over again\nyielding all manner of breakfasts like “bacon with sausage with scrambled eggs\nwith bacon . . . ” We won’t though. This time we’ll pick bread. There are three\nrules for that, each of which contains only a terminal. We’ll pick “English\nmuffin”.

\n

With that, every nonterminal in the string has been expanded until it finally\ncontains only terminals and we’re left with:

\"Playing\" the grammar to generate a string.\n

Throw in some ham and Hollandaise, and you’ve got eggs Benedict.

\n

Any time we hit a rule that had multiple productions, we just picked one\narbitrarily. It is this flexibility that allows a short number of grammar rules\nto encode a combinatorially larger set of strings. The fact that a rule can\nrefer to itself—directly or indirectly—kicks it up even more, letting us\npack an infinite number of strings into a finite grammar.

\n

5 . 1 . 2Enhancing our notation

\n

Stuffing an infinite set of strings in a handful of rules is pretty fantastic,\nbut let’s take it further. Our notation works, but it’s tedious. So, like any\ngood language designer, we’ll sprinkle a little syntactic sugar on top—some\nextra convenience notation. In addition to terminals and nonterminals, we’ll\nallow a few other kinds of expressions in the body of a rule:

\n
    \n
  • \n

    Instead of repeating the rule name each time we want to add another\nproduction for it, we’ll allow a series of productions separated by a pipe\n(|).

    \n
    bread → "toast" | "biscuits" | "English muffin" ;\n
    \n
    • \n
  • \n

    Further, we’ll allow parentheses for grouping and then allow | within that\nto select one from a series of options within the middle of a production.

    \n
    protein → ( "scrambled" | "poached" | "fried" ) "eggs" ;\n
    \n
    • \n
  • \n

    Using recursion to support repeated sequences of symbols has a certain\nappealing purity, but it’s kind of a chore to\nmake a separate named sub-rule each time we want to loop. So, we also use a\npostfix * to allow the previous symbol or group to be repeated zero or\nmore times.

    \n
    crispiness → "really" "really"* ;\n
    \n
    • \n
\n\n
    \n
  • \n

    A postfix + is similar, but requires the preceding production to appear\nat least once.

    \n
    crispiness → "really"+ ;\n
    \n
    • \n
  • \n

    A postfix ? is for an optional production. The thing before it can appear\nzero or one time, but not more.

    \n
    breakfast → protein ( "with" breakfast "on the side" )? ;\n
    \n
    • \n
\n

With all of those syntactic niceties, our breakfast grammar condenses down to:

\n
breakfast → protein ( "with" breakfast "on the side" )?\n          | bread ;\n\nprotein   → "really"+ "crispy" "bacon"\n          | "sausage"\n          | ( "scrambled" | "poached" | "fried" ) "eggs" ;\n\nbread     → "toast" | "biscuits" | "English muffin" ;\n
\n

Not too bad, I hope. If you’re used to grep or using regular\nexpressions in your text editor, most of the punctuation should be\nfamiliar. The main difference is that symbols here represent entire tokens, not\nsingle characters.

\n

We’ll use this notation throughout the rest of the book to precisely describe\nLox’s grammar. As you work on programming languages, you’ll find that\ncontext-free grammars (using this or EBNF or some other notation) help you\ncrystallize your informal syntax design ideas. They are also a handy medium for\ncommunicating with other language hackers about syntax.

\n

The rules and productions we define for Lox are also our guide to the tree data\nstructure we’re going to implement to represent code in memory. Before we can do\nthat, we need an actual grammar for Lox, or at least enough of one for us to get\nstarted.

\n

5 . 1 . 3A Grammar for Lox expressions

\n

In the previous chapter, we did Lox’s entire lexical grammar in one fell swoop.\nEvery keyword and bit of punctuation is there. The syntactic grammar is larger,\nand it would be a real bore to grind through the entire thing before we actually\nget our interpreter up and running.

\n

Instead, we’ll crank through a subset of the language in the next couple of\nchapters. Once we have that mini-language represented, parsed, and interpreted,\nthen later chapters will progressively add new features to it, including the new\nsyntax. For now, we are going to worry about only a handful of expressions:

\n
    \n
  • \n

    Literals. Numbers, strings, Booleans, and nil.

    \n
    • \n
  • \n

    Unary expressions. A prefix ! to perform a logical not, and - to\nnegate a number.

    \n
    • \n
  • \n

    Binary expressions. The infix arithmetic (+, -, *, /) and logic\noperators (==, !=, <, <=, >, >=) we know and love.

    \n
    • \n
  • \n

    Parentheses. A pair of ( and ) wrapped around an expression.

    \n
    • \n
\n

That gives us enough syntax for expressions like:

\n
1 - (2 * 3) < 4 == false\n
\n

Using our handy dandy new notation, here’s a grammar for those:

\n
expression     → literal\n               | unary\n               | binary\n               | grouping ;\n\nliteral        → NUMBER | STRING | "true" | "false" | "nil" ;\ngrouping       → "(" expression ")" ;\nunary          → ( "-" | "!" ) expression ;\nbinary         → expression operator expression ;\noperator       → "==" | "!=" | "<" | "<=" | ">" | ">="\n               | "+"  | "-"  | "*" | "/" ;\n
\n

There’s one bit of extra metasyntax here. In addition\nto quoted strings for terminals that match exact lexemes, we CAPITALIZE\nterminals that are a single lexeme whose text representation may vary. NUMBER\nis any number literal, and STRING is any string literal. Later, we’ll do the\nsame for IDENTIFIER.

\n

This grammar is actually ambiguous, which we’ll see when we get to parsing it.\nBut it’s good enough for now.

\n\n

5 . 2Implementing Syntax Trees

\n

Finally, we get to write some code. That little expression grammar is our\nskeleton. Since the grammar is recursive—note how grouping, unary, and\nbinary all refer back to expression—our data structure will form a tree.\nSince this structure represents the syntax of our language, it’s called a syntax tree.

\n\n

Our scanner used a single Token class to represent all kinds of lexemes. To\ndistinguish the different kinds—think the number 123 versus the string\n\"123\"—we included a simple TokenType enum. Syntax trees are not so homogeneous. Unary expressions have a single operand,\nbinary expressions have two, and literals have none.

\n

We could mush that all together into a single Expression class with an\narbitrary list of children. Some compilers do. But I like getting the most out\nof Java’s type system. So we’ll define a base class for expressions. Then, for\neach kind of expression—each production under expression—we create a\nsubclass that has fields for the nonterminals specific to that rule. This way,\nwe get a compile error if we, say, try to access the second operand of a unary\nexpression.

\n\n

Something like this:

\n
package com.craftinginterpreters.lox;\n\nabstract class Expr { \n  static class Binary extends Expr {\n    Binary(Expr left, Token operator, Expr right) {\n      this.left = left;\n      this.operator = operator;\n      this.right = right;\n    }\n\n    final Expr left;\n    final Token operator;\n    final Expr right;\n  }\n\n  // Other expressions...\n}\n
\n\n

Expr is the base class that all expression classes inherit from. As you can see\nfrom Binary, the subclasses are nested inside of it. There’s no technical need\nfor this, but it lets us cram all of the classes into a single Java file.

\n

5 . 2 . 1Disoriented objects

\n

You’ll note that, much like the Token class, there aren’t any methods here. It’s\na dumb structure. Nicely typed, but merely a bag of data. This feels strange in\nan object-oriented language like Java. Shouldn’t the class do stuff?

\n

The problem is that these tree classes aren’t owned by any single domain. Should\nthey have methods for parsing since that’s where the trees are created? Or\ninterpreting since that’s where they are consumed? Trees span the border between\nthose territories, which means they are really owned by neither.

\n

In fact, these types exist to enable the parser and interpreter to\ncommunicate. That lends itself to types that are simply data with no\nassociated behavior. This style is very natural in functional languages like\nLisp and ML where all data is separate from behavior, but it feels odd in\nJava.

\n

Functional programming aficionados right now are jumping up to exclaim “See!\nObject-oriented languages are a bad fit for an interpreter!” I won’t go that\nfar. You’ll recall that the scanner itself was admirably suited to\nobject-orientation. It had all of the mutable state to keep track of where it\nwas in the source code, a well-defined set of public methods, and a handful of\nprivate helpers.

\n

My feeling is that each phase or part of the interpreter works fine in an\nobject-oriented style. It is the data structures that flow between them that are\nstripped of behavior.

\n

5 . 2 . 2Metaprogramming the trees

\n

Java can express behavior-less classes, but I wouldn’t say that it’s\nparticularly great at it. Eleven lines of code to stuff three fields in an\nobject is pretty tedious, and when we’re all done, we’re going to have 21 of\nthese classes.

\n

I don’t want to waste your time or my ink writing all that down. Really, what is\nthe essence of each subclass? A name, and a list of typed fields. That’s it.\nWe’re smart language hackers, right? Let’s automate.

\n\n

Instead of tediously handwriting each class definition, field declaration,\nconstructor, and initializer, we’ll hack together a script that does it for us. It has a description of each\ntree type—its name and fields—and it prints out the Java code needed to\ndefine a class with that name and state.

\n

This script is a tiny Java command-line app that generates a file named\n“Expr.java”:

\n\n
tool/GenerateAst.java
\ncreate new file
\n
package com.craftinginterpreters.tool;\n\nimport java.io.IOException;\nimport java.io.PrintWriter;\nimport java.util.Arrays;\nimport java.util.List;\n\npublic class GenerateAst {\n  public static void main(String[] args) throws IOException {\n    if (args.length != 1) {\n      System.err.println("Usage: generate_ast <output directory>");\n      System.exit(64);\n    }\n    String outputDir = args[0];\n  }\n}\n
\n
tool/GenerateAst.java, create new file
\n\n

Note that this file is in a different package, .tool instead of .lox. This\nscript isn’t part of the interpreter itself. It’s a tool we, the people\nhacking on the interpreter, run ourselves to generate the syntax tree classes.\nWhen it’s done, we treat “Expr.java” like any other file in the implementation.\nWe are merely automating how that file gets authored.

\n

To generate the classes, it needs to have some description of each type and its\nfields.

\n
    String outputDir = args[0];\n
tool/GenerateAst.java
\nin main()
\n
    defineAst(outputDir, "Expr", Arrays.asList(\n      "Binary   : Expr left, Token operator, Expr right",\n      "Grouping : Expr expression",\n      "Literal  : Object value",\n      "Unary    : Token operator, Expr right"\n    ));\n
  }\n
\n
tool/GenerateAst.java, in main()
\n\n

For brevity’s sake, I jammed the descriptions of the expression types into\nstrings. Each is the name of the class followed by : and the list of fields,\nseparated by commas. Each field has a type and a name.

\n

The first thing defineAst() needs to do is output the base Expr class.

\n
tool/GenerateAst.java
\nadd after main()
\n
  private static void defineAst(\n      String outputDir, String baseName, List<String> types)\n      throws IOException {\n    String path = outputDir + "/" + baseName + ".java";\n    PrintWriter writer = new PrintWriter(path, "UTF-8");\n\n    writer.println("package com.craftinginterpreters.lox;");\n    writer.println();\n    writer.println("import java.util.List;");\n    writer.println();\n    writer.println("abstract class " + baseName + " {");\n\n    writer.println("}");\n    writer.close();\n  }\n
\n
tool/GenerateAst.java, add after main()
\n\n

When we call this, baseName is “Expr”, which is both the name of the class and\nthe name of the file it outputs. We pass this as an argument instead of\nhardcoding the name because we’ll add a separate family of classes later for\nstatements.

\n

Inside the base class, we define each subclass.

\n
    writer.println("abstract class " + baseName + " {");\n\n
tool/GenerateAst.java
\nin defineAst()
\n
    // The AST classes.\n    for (String type : types) {\n      String className = type.split(":")[0].trim();\n      String fields = type.split(":")[1].trim(); \n      defineType(writer, baseName, className, fields);\n    }\n
    writer.println("}");\n
\n
tool/GenerateAst.java, in defineAst()
\n\n\n

That code, in turn, calls:

\n
tool/GenerateAst.java
\nadd after defineAst()
\n
  private static void defineType(\n      PrintWriter writer, String baseName,\n      String className, String fieldList) {\n    writer.println("  static class " + className + " extends " +\n        baseName + " {");\n\n    // Constructor.\n    writer.println("    " + className + "(" + fieldList + ") {");\n\n    // Store parameters in fields.\n    String[] fields = fieldList.split(", ");\n    for (String field : fields) {\n      String name = field.split(" ")[1];\n      writer.println("      this." + name + " = " + name + ";");\n    }\n\n    writer.println("    }");\n\n    // Fields.\n    writer.println();\n    for (String field : fields) {\n      writer.println("    final " + field + ";");\n    }\n\n    writer.println("  }");\n  }\n
\n
tool/GenerateAst.java, add after defineAst()
\n\n

There we go. All of that glorious Java boilerplate is done. It declares each\nfield in the class body. It defines a constructor for the class with parameters\nfor each field and initializes them in the body.

\n

Compile and run this Java program now and it blasts\nout a new “.java” file containing a few dozen lines of code. That file’s\nabout to get even longer.

\n\n

5 . 3Working with Trees

\n

Put on your imagination hat for a moment. Even though we aren’t there yet,\nconsider what the interpreter will do with the syntax trees. Each kind of\nexpression in Lox behaves differently at runtime. That means the interpreter\nneeds to select a different chunk of code to handle each expression type. With\ntokens, we can simply switch on the TokenType. But we don’t have a “type” enum\nfor the syntax trees, just a separate Java class for each one.

\n

We could write a long chain of type tests:

\n
if (expr instanceof Expr.Binary) {\n  // ...\n} else if (expr instanceof Expr.Grouping) {\n  // ...\n} else // ...\n
\n

But all of those sequential type tests are slow. Expression types whose names\nare alphabetically later would take longer to execute because they’d fall\nthrough more if cases before finding the right type. That’s not my idea of an\nelegant solution.

\n

We have a family of classes and we need to associate a chunk of behavior with\neach one. The natural solution in an object-oriented language like Java is to\nput those behaviors into methods on the classes themselves. We could add an\nabstract interpret() method on Expr\nwhich each subclass would then implement to interpret itself.

\n\n

This works alright for tiny projects, but it scales poorly. Like I noted before,\nthese tree classes span a few domains. At the very least, both the parser and\ninterpreter will mess with them. As you’ll see later, we need to\ndo name resolution on them. If our language was statically typed, we’d have a\ntype checking pass.

\n

If we added instance methods to the expression classes for every one of those\noperations, that would smush a bunch of different domains together. That\nviolates separation of concerns and leads to hard-to-maintain code.

\n

5 . 3 . 1The expression problem

\n

This problem is more fundamental than it may seem at first. We have a handful of\ntypes, and a handful of high-level operations like “interpret”. For each pair of\ntype and operation, we need a specific implementation. Picture a table:

\"A\n

Rows are types, and columns are operations. Each cell represents the unique\npiece of code to implement that operation on that type.

\n

An object-oriented language like Java assumes that all of the code in one row\nnaturally hangs together. It figures all the things you do with a type are\nlikely related to each other, and the language makes it easy to define them\ntogether as methods inside the same class.

\"The\n

This makes it easy to extend the table by adding new rows. Simply define a new\nclass. No existing code has to be touched. But imagine if you want to add a new\noperation—a new column. In Java, that means cracking open each of those\nexisting classes and adding a method to it.

\n

Functional paradigm languages in the ML family flip that\naround. There, you don’t have classes with methods. Types and functions are\ntotally distinct. To implement an operation for a number of different types, you\ndefine a single function. In the body of that function, you use pattern\nmatching—sort of a type-based switch on steroids—to implement the\noperation for each type all in one place.

\n\n

This makes it trivial to add new operations—simply define another function\nthat pattern matches on all of the types.

\"The\n

But, conversely, adding a new type is hard. You have to go back and add a new\ncase to all of the pattern matches in all of the existing functions.

\n

Each style has a certain “grain” to it. That’s what the paradigm name literally\nsays—an object-oriented language wants you to orient your code along the\nrows of types. A functional language instead encourages you to lump each\ncolumn’s worth of code together into a function.

\n

A bunch of smart language nerds noticed that neither style made it easy to add\nboth rows and columns to the table. They called this\ndifficulty the “expression problem” because—like we are now—they first ran\ninto it when they were trying to figure out the best way to model expression\nsyntax tree nodes in a compiler.

\n\n

People have thrown all sorts of language features, design patterns, and\nprogramming tricks to try to knock that problem down but no perfect language has\nfinished it off yet. In the meantime, the best we can do is try to pick a\nlanguage whose orientation matches the natural architectural seams in the\nprogram we’re writing.

\n

Object-orientation works fine for many parts of our interpreter, but these tree\nclasses rub against the grain of Java. Fortunately, there’s a design pattern we\ncan bring to bear on it.

\n

5 . 3 . 2The Visitor pattern

\n

The Visitor pattern is the most widely misunderstood pattern in all of\nDesign Patterns, which is really saying something when you look at the\nsoftware architecture excesses of the past couple of decades.

\n

The trouble starts with terminology. The pattern isn’t about “visiting”, and the\n“accept” method in it doesn’t conjure up any helpful imagery either. Many think\nthe pattern has to do with traversing trees, which isn’t the case at all. We\nare going to use it on a set of classes that are tree-like, but that’s a\ncoincidence. As you’ll see, the pattern works as well on a single object.

\n

The Visitor pattern is really about approximating the functional style within an\nOOP language. It lets us add new columns to that table easily. We can define all\nof the behavior for a new operation on a set of types in one place, without\nhaving to touch the types themselves. It does this the same way we solve almost\nevery problem in computer science: by adding a layer of indirection.

\n

Before we apply it to our auto-generated Expr classes, let’s walk through a\nsimpler example. Say we have two kinds of pastries: beignets and crullers.

\n\n
  abstract class Pastry {\n  }\n\n  class Beignet extends Pastry {\n  }\n\n  class Cruller extends Pastry {\n  }\n
\n\n

We want to be able to define new pastry operations—cooking them, eating them,\ndecorating them, etc.—without having to add a new method to each class every\ntime. Here’s how we do it. First, we define a separate interface.

\n
  interface PastryVisitor {\n    void visitBeignet(Beignet beignet); \n    void visitCruller(Cruller cruller);\n  }\n
\n\n\n

Each operation that can be performed on pastries is a new class that implements\nthat interface. It has a concrete method for each type of pastry. That keeps the\ncode for the operation on both types all nestled snugly together in one class.

\n

Given some pastry, how do we route it to the correct method on the visitor based\non its type? Polymorphism to the rescue! We add this method to Pastry:

\n
  abstract class Pastry {\n
    abstract void accept(PastryVisitor visitor);\n
  }\n
\n\n

Each subclass implements it.

\n
  class Beignet extends Pastry {\n
    @Override\n    void accept(PastryVisitor visitor) {\n      visitor.visitBeignet(this);\n    }\n
  }\n
\n\n

And:

\n
  class Cruller extends Pastry {\n
    @Override\n    void accept(PastryVisitor visitor) {\n      visitor.visitCruller(this);\n    }\n
  }\n
\n\n

To perform an operation on a pastry, we call its accept() method and pass in\nthe visitor for the operation we want to execute. The pastry—the specific\nsubclass’s overriding implementation of accept()—turns around and calls the\nappropriate visit method on the visitor and passes itself to it.

\n

That’s the heart of the trick right there. It lets us use polymorphic dispatch\non the pastry classes to select the appropriate method on the visitor class.\nIn the table, each pastry class is a row, but if you look at all of the methods\nfor a single visitor, they form a column.

\"Now\n

We added one accept() method to each class, and we can use it for as many\nvisitors as we want without ever having to touch the pastry classes again. It’s\na clever pattern.

\n

5 . 3 . 3Visitors for expressions

\n

OK, let’s weave it into our expression classes. We’ll also refine the pattern a little. In the pastry example, the\nvisit and accept() methods don’t return anything. In practice, visitors often\nwant to define operations that produce values. But what return type should\naccept() have? We can’t assume every visitor class wants to produce the same\ntype, so we’ll use generics to let each implementation fill in a return type.

\n\n

First, we define the visitor interface. Again, we nest it inside the base class\nso that we can keep everything in one file.

\n
    writer.println("abstract class " + baseName + " {");\n\n
tool/GenerateAst.java
\nin defineAst()
\n
    defineVisitor(writer, baseName, types);\n\n
    // The AST classes.\n
\n
tool/GenerateAst.java, in defineAst()
\n\n

That function generates the visitor interface.

\n
tool/GenerateAst.java
\nadd after defineAst()
\n
  private static void defineVisitor(\n      PrintWriter writer, String baseName, List<String> types) {\n    writer.println("  interface Visitor<R> {");\n\n    for (String type : types) {\n      String typeName = type.split(":")[0].trim();\n      writer.println("    R visit" + typeName + baseName + "(" +\n          typeName + " " + baseName.toLowerCase() + ");");\n    }\n\n    writer.println("  }");\n  }\n
\n
tool/GenerateAst.java, add after defineAst()
\n\n

Here, we iterate through all of the subclasses and declare a visit method for\neach one. When we define new expression types later, this will automatically\ninclude them.

\n

Inside the base class, we define the abstract accept() method.

\n
      defineType(writer, baseName, className, fields);\n    }\n
tool/GenerateAst.java
\nin defineAst()
\n
\n\n    // The base accept() method.\n    writer.println();\n    writer.println("  abstract <R> R accept(Visitor<R> visitor);");\n\n
    writer.println("}");\n
\n
tool/GenerateAst.java, in defineAst()
\n\n

Finally, each subclass implements that and calls the right visit method for its\nown type.

\n
    writer.println("    }");\n
tool/GenerateAst.java
\nin defineType()
\n
\n\n    // Visitor pattern.\n    writer.println();\n    writer.println("    @Override");\n    writer.println("    <R> R accept(Visitor<R> visitor) {");\n    writer.println("      return visitor.visit" +\n        className + baseName + "(this);");\n    writer.println("    }");\n
\n\n    // Fields.\n
\n
tool/GenerateAst.java, in defineType()
\n\n

There we go. Now we can define operations on expressions without having to muck\nwith the classes or our generator script. Compile and run this generator script\nto output an updated “Expr.java” file. It contains a generated Visitor\ninterface and a set of expression node classes that support the Visitor pattern\nusing it.

\n

Before we end this rambling chapter, let’s implement that Visitor interface and\nsee the pattern in action.

\n

5 . 4A (Not Very) Pretty Printer

\n

When we debug our parser and interpreter, it’s often useful to look at a parsed\nsyntax tree and make sure it has the structure we expect. We could inspect it in\nthe debugger, but that can be a chore.

\n

Instead, we’d like some code that, given a syntax tree, produces an unambiguous\nstring representation of it. Converting a tree to a string is sort of the\nopposite of a parser, and is often called “pretty printing” when the goal is to\nproduce a string of text that is valid syntax in the source language.

\n

That’s not our goal here. We want the string to very explicitly show the nesting\nstructure of the tree. A printer that returned 1 + 2 * 3 isn’t super helpful\nif what we’re trying to debug is whether operator precedence is handled\ncorrectly. We want to know if the + or * is at the top of the tree.

\n

To that end, the string representation we produce isn’t going to be Lox syntax.\nInstead, it will look a lot like, well, Lisp. Each expression is explicitly\nparenthesized, and all of its subexpressions and tokens are contained in that.

\n

Given a syntax tree like:

\"An\n

It produces:

\n
(* (- 123) (group 45.67))\n
\n

Not exactly “pretty”, but it does show the nesting and grouping explicitly. To\nimplement this, we define a new class.

\n
lox/AstPrinter.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nclass AstPrinter implements Expr.Visitor<String> {\n  String print(Expr expr) {\n    return expr.accept(this);\n  }\n}\n
\n
lox/AstPrinter.java, create new file
\n\n

As you can see, it implements the visitor interface. That means we need visit\nmethods for each of the expression types we have so far.

\n
    return expr.accept(this);\n  }\n
lox/AstPrinter.java
\nadd after print()
\n
\n\n  @Override\n  public String visitBinaryExpr(Expr.Binary expr) {\n    return parenthesize(expr.operator.lexeme,\n                        expr.left, expr.right);\n  }\n\n  @Override\n  public String visitGroupingExpr(Expr.Grouping expr) {\n    return parenthesize("group", expr.expression);\n  }\n\n  @Override\n  public String visitLiteralExpr(Expr.Literal expr) {\n    if (expr.value == null) return "nil";\n    return expr.value.toString();\n  }\n\n  @Override\n  public String visitUnaryExpr(Expr.Unary expr) {\n    return parenthesize(expr.operator.lexeme, expr.right);\n  }\n
}\n
\n
lox/AstPrinter.java, add after print()
\n\n

Literal expressions are easy—they convert the value to a string with a little\ncheck to handle Java’s null standing in for Lox’s nil. The other expressions\nhave subexpressions, so they use this parenthesize() helper method:

\n
lox/AstPrinter.java
\nadd after visitUnaryExpr()
\n
  private String parenthesize(String name, Expr... exprs) {\n    StringBuilder builder = new StringBuilder();\n\n    builder.append("(").append(name);\n    for (Expr expr : exprs) {\n      builder.append(" ");\n      builder.append(expr.accept(this));\n    }\n    builder.append(")");\n\n    return builder.toString();\n  }\n
\n
lox/AstPrinter.java, add after visitUnaryExpr()
\n\n

It takes a name and a list of subexpressions and wraps them all up in\nparentheses, yielding a string like:

\n
(+ 1 2)\n
\n

Note that it calls accept() on each subexpression and passes in itself. This\nis the recursive step that lets us print an entire\ntree.

\n\n

We don’t have a parser yet, so it’s hard to see this in action. For now, we’ll\nhack together a little main() method that manually instantiates a tree and\nprints it.

\n
lox/AstPrinter.java
\nadd after parenthesize()
\n
  public static void main(String[] args) {\n    Expr expression = new Expr.Binary(\n        new Expr.Unary(\n            new Token(TokenType.MINUS, "-", null, 1),\n            new Expr.Literal(123)),\n        new Token(TokenType.STAR, "*", null, 1),\n        new Expr.Grouping(\n            new Expr.Literal(45.67)));\n\n    System.out.println(new AstPrinter().print(expression));\n  }\n
\n
lox/AstPrinter.java, add after parenthesize()
\n\n

If we did everything right, it prints:

\n
(* (- 123) (group 45.67))\n
\n

You can go ahead and delete this method. We won’t need it. Also, as we add new\nsyntax tree types, I won’t bother showing the necessary visit methods for them\nin AstPrinter. If you want to (and you want the Java compiler to not yell at\nyou), go ahead and add them yourself. It will come in handy in the next chapter\nwhen we start parsing Lox code into syntax trees. Or, if you don’t care to\nmaintain AstPrinter, feel free to delete it. We won’t need it again.

\n
\n

Challenges

\n
    \n
  1. \n

    Earlier, I said that the |, *, and + forms we added to our grammar\nmetasyntax were just syntactic sugar. Take this grammar:

    \n
    expr → expr ( "(" ( expr ( "," expr )* )? ")" | "." IDENTIFIER )+\n     | IDENTIFIER\n     | NUMBER\n
    \n

    Produce a grammar that matches the same language but does not use any of\nthat notational sugar.

    \n

    Bonus: What kind of expression does this bit of grammar encode?

    \n
    2. \n
  2. \n

    The Visitor pattern lets you emulate the functional style in an\nobject-oriented language. Devise a complementary pattern for a functional\nlanguage. It should let you bundle all of the operations on one type\ntogether and let you define new types easily.

    \n

    (SML or Haskell would be ideal for this exercise, but Scheme or another Lisp\nworks as well.)

    \n
    3. \n
  3. \n

    In reverse Polish notation (RPN), the operands to an arithmetic\noperator are both placed before the operator, so 1 + 2 becomes 1 2 +.\nEvaluation proceeds from left to right. Numbers are pushed onto an implicit\nstack. An arithmetic operator pops the top two numbers, performs the\noperation, and pushes the result. Thus, this:

    \n
    (1 + 2) * (4 - 3)\n
    \n

    in RPN becomes:

    \n
    1 2 + 4 3 - *\n
    \n

    Define a visitor class for our syntax tree classes that takes an expression,\nconverts it to RPN, and returns the resulting string.

    \n
    4. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/resolving-and-binding.html", "content": "\n\n\n\nResolving and Binding · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
11
\n

Resolving and Binding

\n\n
\n

Once in a while you find yourself in an odd situation. You get into it by\ndegrees and in the most natural way but, when you are right in the midst of\nit, you are suddenly astonished and ask yourself how in the world it all came\nabout.

\n

Thor Heyerdahl, Kon-Tiki

\n
\n

Oh, no! Our language implementation is taking on water! Way back when we added\nvariables and blocks, we had scoping nice and tight. But when we\nlater added closures, a hole opened in our formerly waterproof\ninterpreter. Most real programs are unlikely to slip through this hole, but as\nlanguage implementers, we take a sacred vow to care about correctness even in\nthe deepest, dampest corners of the semantics.

\n

We will spend this entire chapter exploring that leak, and then carefully\npatching it up. In the process, we will gain a more rigorous understanding of\nlexical scoping as used by Lox and other languages in the C tradition. We’ll\nalso get a chance to learn about semantic analysis—a powerful technique for\nextracting meaning from the user’s source code without having to run it.

\n

11 . 1Static Scope

\n

A quick refresher: Lox, like most modern languages, uses lexical scoping. This\nmeans that you can figure out which declaration a variable name refers to just\nby reading the text of the program. For example:

\n
var a = "outer";\n{\n  var a = "inner";\n  print a;\n}\n
\n

Here, we know that the a being printed is the variable declared on the\nprevious line, and not the global one. Running the program doesn’t—can’t—affect this. The scope rules are part of the static semantics of the language,\nwhich is why they’re also called static scope.

\n

I haven’t spelled out those scope rules, but now is the time for precision:

\n\n

A variable usage refers to the preceding declaration with the same name in the\ninnermost scope that encloses the expression where the variable is used.

\n

There’s a lot to unpack in that:

\n
    \n
  • \n

    I say “variable usage” instead of “variable expression” to cover both\nvariable expressions and assignments. Likewise with “expression where the\nvariable is used”.

    \n
    • \n
  • \n

    “Preceding” means appearing before in the program text.

    \n
    var a = "outer";\n{\n  print a;\n  var a = "inner";\n}\n
    \n

    Here, the a being printed is the outer one since it appears before the print statement that uses it. In most\ncases, in straight line code, the declaration preceding in text will also\nprecede the usage in time. But that’s not always true. As we’ll see,\nfunctions may defer a chunk of code such that its dynamic temporal\nexecution no longer mirrors the static textual ordering.

    \n
    • \n
  • \n

    “Innermost” is there because of our good friend shadowing. There may be more\nthan one variable with the given name in enclosing scopes, as in:

    \n
    var a = "outer";\n{\n  var a = "inner";\n  print a;\n}\n
    \n

    Our rule disambiguates this case by saying the innermost scope wins.

    \n
    • \n
\n

Since this rule makes no mention of any runtime behavior, it implies that a\nvariable expression always refers to the same declaration through the entire\nexecution of the program. Our interpreter so far mostly implements the rule\ncorrectly. But when we added closures, an error snuck in.

\n
var a = "global";\n{\n  fun showA() {\n    print a;\n  }\n\n  showA();\n  var a = "block";\n  showA();\n}\n
\n

Before you type this in and run it, decide what you\nthink it should print.

\n\n

OK . . . got it? If you’re familiar with closures in other languages, you’ll expect\nit to print “global” twice. The first call to showA() should definitely print\n“global” since we haven’t even reached the declaration of the inner a yet. And\nby our rule that a variable expression always resolves to the same variable,\nthat implies the second call to showA() should print the same thing.

\n

Alas, it prints:

\n
global\nblock\n
\n

Let me stress that this program never reassigns any variable and contains only a\nsingle print statement. Yet, somehow, that print statement for a\nnever-assigned variable prints two different values at different points in time.\nWe definitely broke something somewhere.

\n

11 . 1 . 1Scopes and mutable environments

\n

In our interpreter, environments are the dynamic manifestation of static scopes.\nThe two mostly stay in sync with each other—we create a new environment when\nwe enter a new scope, and discard it when we leave the scope. There is one other\noperation we perform on environments: binding a variable in one. This is where\nour bug lies.

\n

Let’s walk through that problematic example and see what the environments look\nlike at each step. First, we declare a in the global scope.

\"The\n

That gives us a single environment with a single variable in it. Then we enter\nthe block and execute the declaration of showA().

\"A\n

We get a new environment for the block. In that, we declare one name, showA,\nwhich is bound to the LoxFunction object we create to represent the function.\nThat object has a closure field that captures the environment where the\nfunction was declared, so it has a reference back to the environment for the\nblock.

\n

Now we call showA().

\"An\n

The interpreter dynamically creates a new environment for the function body of\nshowA(). It’s empty since that function doesn’t declare any variables. The\nparent of that environment is the function’s closure—the outer block\nenvironment.

\n

Inside the body of showA(), we print the value of a. The interpreter looks\nup this value by walking the chain of environments. It gets all the way\nto the global environment before finding it there and printing \"global\".\nGreat.

\n

Next, we declare the second a, this time inside the block.

\"The\n

It’s in the same block—the same scope—as showA(), so it goes into the\nsame environment, which is also the same environment showA()’s closure refers\nto. This is where it gets interesting. We call showA() again.

\"An\n

We create a new empty environment for the body of showA() again, wire it up to\nthat closure, and run the body. When the interpreter walks the chain of\nenvironments to find a, it now discovers the new a in the block\nenvironment. Boo.

\n

I chose to implement environments in a way that I hoped would agree with your\ninformal intuition around scopes. We tend to consider all of the code within a\nblock as being within the same scope, so our interpreter uses a single\nenvironment to represent that. Each environment is a mutable hash table. When a\nnew local variable is declared, it gets added to the existing environment for\nthat scope.

\n

That intuition, like many in life, isn’t quite right. A block is not necessarily\nall the same scope. Consider:

\n
{\n  var a;\n  // 1.\n  var b;\n  // 2.\n}\n
\n

At the first marked line, only a is in scope. At the second line, both a and\nb are. If you define a “scope” to be a set of declarations, then those are\nclearly not the same scope—they don’t contain the same declarations. It’s\nlike each var statement splits the block into two\nseparate scopes, the scope before the variable is declared and the one after,\nwhich includes the new variable.

\n\n

But in our implementation, environments do act like the entire block is one\nscope, just a scope that changes over time. Closures do not like that. When a\nfunction is declared, it captures a reference to the current environment. The\nfunction should capture a frozen snapshot of the environment as it existed at\nthe moment the function was declared. But instead, in the Java code, it has a\nreference to the actual mutable environment object. When a variable is later\ndeclared in the scope that environment corresponds to, the closure sees the new\nvariable, even though the declaration does not precede the function.

\n

11 . 1 . 2Persistent environments

\n

There is a style of programming that uses what are called persistent data\nstructures. Unlike the squishy data structures you’re familiar with in\nimperative programming, a persistent data structure can never be directly\nmodified. Instead, any “modification” to an existing structure produces a brand new object that contains all of the original data and\nthe new modification. The original is left unchanged.

\n\n

If we were to apply that technique to Environment, then every time you declared\na variable it would return a new environment that contained all of the\npreviously declared variables along with the one new name. Declaring a variable\nwould do the implicit “split” where you have an environment before the variable\nis declared and one after:

\"Separate\n

A closure retains a reference to the Environment instance in play when the\nfunction was declared. Since any later declarations in that block would produce\nnew Environment objects, the closure wouldn’t see the new variables and our bug\nwould be fixed.

\n

This is a legit way to solve the problem, and it’s the classic way to implement\nenvironments in Scheme interpreters. We could do that for Lox, but it would mean\ngoing back and changing a pile of existing code.

\n

I won’t drag you through that. We’ll keep the way we represent environments the\nsame. Instead of making the data more statically structured, we’ll bake the\nstatic resolution into the access operation itself.

\n

11 . 2Semantic Analysis

\n

Our interpreter resolves a variable—tracks down which declaration it\nrefers to—each and every time the variable expression is evaluated. If that\nvariable is swaddled inside a loop that runs a thousand times, that variable\ngets re-resolved a thousand times.

\n

We know static scope means that a variable usage always resolves to the same\ndeclaration, which can be determined just by looking at the text. Given that,\nwhy are we doing it dynamically every time? Doing so doesn’t just open the hole\nthat leads to our annoying bug, it’s also needlessly slow.

\n

A better solution is to resolve each variable use once. Write a chunk of code\nthat inspects the user’s program, finds every variable mentioned, and figures\nout which declaration each refers to. This process is an example of a semantic\nanalysis. Where a parser tells only if a program is grammatically correct (a\nsyntactic analysis), semantic analysis goes farther and starts to figure out\nwhat pieces of the program actually mean. In this case, our analysis will\nresolve variable bindings. We’ll know not just that an expression is a\nvariable, but which variable it is.

\n

There are a lot of ways we could store the binding between a variable and its\ndeclaration. When we get to the C interpreter for Lox, we’ll have a much more\nefficient way of storing and accessing local variables. But for jlox, I want to\nminimize the collateral damage we inflict on our existing codebase. I’d hate to\nthrow out a bunch of mostly fine code.

\n

Instead, we’ll store the resolution in a way that makes the most out of our\nexisting Environment class. Recall how the accesses of a are interpreted in\nthe problematic example.

\"An\n

In the first (correct) evaluation, we look at three environments in the chain\nbefore finding the global declaration of a. Then, when the inner a is later\ndeclared in a block scope, it shadows the global one.

\"An\n

The next lookup walks the chain, finds a in the second environment and\nstops there. Each environment corresponds to a single lexical scope where\nvariables are declared. If we could ensure a variable lookup always walked the\nsame number of links in the environment chain, that would ensure that it\nfound the same variable in the same scope every time.

\n

To “resolve” a variable usage, we only need to calculate how many “hops” away\nthe declared variable will be in the environment chain. The interesting question\nis when to do this calculation—or, put differently, where in our\ninterpreter’s implementation do we stuff the code for it?

\n

Since we’re calculating a static property based on the structure of the source\ncode, the obvious answer is in the parser. That is the traditional home, and is\nwhere we’ll put it later in clox. It would work here too, but I want an excuse to\nshow you another technique. We’ll write our resolver as a separate pass.

\n

11 . 2 . 1A variable resolution pass

\n

After the parser produces the syntax tree, but before the interpreter starts\nexecuting it, we’ll do a single walk over the tree to resolve all of the\nvariables it contains. Additional passes between parsing and execution are\ncommon. If Lox had static types, we could slide a type checker in there.\nOptimizations are often implemented in separate passes like this too. Basically,\nany work that doesn’t rely on state that’s only available at runtime can be done\nin this way.

\n

Our variable resolution pass works like a sort of mini-interpreter. It walks the\ntree, visiting each node, but a static analysis is different from a dynamic\nexecution:

\n
    \n
  • \n

    There are no side effects. When the static analysis visits a print\nstatement, it doesn’t actually print anything. Calls to native functions or\nother operations that reach out to the outside world are stubbed out and\nhave no effect.

    \n
    • \n
  • \n

    There is no control flow. Loops are visited only once. Both branches are visited in if statements. Logic\noperators are not short-circuited.

    \n
    • \n
\n\n

11 . 3A Resolver Class

\n

Like everything in Java, our variable resolution pass is embodied in a class.

\n
lox/Resolver.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Stack;\n\nclass Resolver implements Expr.Visitor<Void>, Stmt.Visitor<Void> {\n  private final Interpreter interpreter;\n\n  Resolver(Interpreter interpreter) {\n    this.interpreter = interpreter;\n  }\n}\n
\n
lox/Resolver.java, create new file
\n\n

Since the resolver needs to visit every node in the syntax tree, it implements\nthe visitor abstraction we already have in place. Only a few kinds of nodes are\ninteresting when it comes to resolving variables:

\n
    \n
  • \n

    A block statement introduces a new scope for the statements it contains.

    \n
    • \n
  • \n

    A function declaration introduces a new scope for its body and binds its\nparameters in that scope.

    \n
    • \n
  • \n

    A variable declaration adds a new variable to the current scope.

    \n
    • \n
  • \n

    Variable and assignment expressions need to have their variables resolved.

    \n
    • \n
\n

The rest of the nodes don’t do anything special, but we still need to implement\nvisit methods for them that traverse into their subtrees. Even though a +\nexpression doesn’t itself have any variables to resolve, either of its\noperands might.

\n

11 . 3 . 1Resolving blocks

\n

We start with blocks since they create the local scopes where all the magic\nhappens.

\n
lox/Resolver.java
\nadd after Resolver()
\n
  @Override\n  public Void visitBlockStmt(Stmt.Block stmt) {\n    beginScope();\n    resolve(stmt.statements);\n    endScope();\n    return null;\n  }\n
\n
lox/Resolver.java, add after Resolver()
\n\n

This begins a new scope, traverses into the statements inside the block, and\nthen discards the scope. The fun stuff lives in those helper methods. We start\nwith the simple one.

\n
lox/Resolver.java
\nadd after Resolver()
\n
  void resolve(List<Stmt> statements) {\n    for (Stmt statement : statements) {\n      resolve(statement);\n    }\n  }\n
\n
lox/Resolver.java, add after Resolver()
\n\n

This walks a list of statements and resolves each one. It in turn calls:

\n
lox/Resolver.java
\nadd after visitBlockStmt()
\n
  private void resolve(Stmt stmt) {\n    stmt.accept(this);\n  }\n
\n
lox/Resolver.java, add after visitBlockStmt()
\n\n

While we’re at it, let’s add another overload that we’ll need later for\nresolving an expression.

\n
lox/Resolver.java
\nadd after resolve(Stmt stmt)
\n
  private void resolve(Expr expr) {\n    expr.accept(this);\n  }\n
\n
lox/Resolver.java, add after resolve(Stmt stmt)
\n\n

These methods are similar to the evaluate() and execute() methods in\nInterpreter—they turn around and apply the Visitor pattern to the given\nsyntax tree node.

\n

The real interesting behavior is around scopes. A new block scope is created\nlike so:

\n
lox/Resolver.java
\nadd after resolve()
\n
  private void beginScope() {\n    scopes.push(new HashMap<String, Boolean>());\n  }\n
\n
lox/Resolver.java, add after resolve()
\n\n

Lexical scopes nest in both the interpreter and the resolver. They behave like a\nstack. The interpreter implements that stack using a linked list—the chain of\nEnvironment objects. In the resolver, we use an actual Java Stack.

\n
  private final Interpreter interpreter;\n
lox/Resolver.java
\nin class Resolver
\n
  private final Stack<Map<String, Boolean>> scopes = new Stack<>();\n
\n\n  Resolver(Interpreter interpreter) {\n
\n
lox/Resolver.java, in class Resolver
\n\n

This field keeps track of the stack of scopes currently, uh, in scope. Each\nelement in the stack is a Map representing a single block scope. Keys, as in\nEnvironment, are variable names. The values are Booleans, for a reason I’ll\nexplain soon.

\n

The scope stack is only used for local block scopes. Variables declared at the\ntop level in the global scope are not tracked by the resolver since they are\nmore dynamic in Lox. When resolving a variable, if we can’t find it in the stack\nof local scopes, we assume it must be global.

\n

Since scopes are stored in an explicit stack, exiting one is straightforward.

\n
lox/Resolver.java
\nadd after beginScope()
\n
  private void endScope() {\n    scopes.pop();\n  }\n
\n
lox/Resolver.java, add after beginScope()
\n\n

Now we can push and pop a stack of empty scopes. Let’s put some things in them.

\n

11 . 3 . 2Resolving variable declarations

\n

Resolving a variable declaration adds a new entry to the current innermost\nscope’s map. That seems simple, but there’s a little dance we need to do.

\n
lox/Resolver.java
\nadd after visitBlockStmt()
\n
  @Override\n  public Void visitVarStmt(Stmt.Var stmt) {\n    declare(stmt.name);\n    if (stmt.initializer != null) {\n      resolve(stmt.initializer);\n    }\n    define(stmt.name);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitBlockStmt()
\n\n

We split binding into two steps, declaring then defining, in order to handle\nfunny edge cases like this:

\n
var a = "outer";\n{\n  var a = a;\n}\n
\n

What happens when the initializer for a local variable refers to a variable with\nthe same name as the variable being declared? We have a few options:

\n
    \n
  1. \n

    Run the initializer, then put the new variable in scope. Here, the new\nlocal a would be initialized with “outer”, the value of the global one.\nIn other words, the previous declaration would desugar to:

    \n
    var temp = a; // Run the initializer.\nvar a;        // Declare the variable.\na = temp;     // Initialize it.\n
    \n
    2. \n
  2. \n

    Put the new variable in scope, then run the initializer. This means you\ncould observe a variable before it’s initialized, so we would need to figure\nout what value it would have then. Probably nil. That means the new local\na would be re-initialized to its own implicitly initialized value, nil.\nNow the desugaring would look like:

    \n
    var a; // Define the variable.\na = a; // Run the initializer.\n
    \n
    3. \n
  3. \n

    Make it an error to reference a variable in its initializer. Have the\ninterpreter fail either at compile time or runtime if an initializer\nmentions the variable being initialized.

    \n
    4. \n
\n

Do either of those first two options look like something a user actually\nwants? Shadowing is rare and often an error, so initializing a shadowing\nvariable based on the value of the shadowed one seems unlikely to be deliberate.

\n

The second option is even less useful. The new variable will always have the\nvalue nil. There is never any point in mentioning it by name. You could use an\nexplicit nil instead.

\n

Since the first two options are likely to mask user errors, we’ll take the\nthird. Further, we’ll make it a compile error instead of a runtime one. That\nway, the user is alerted to the problem before any code is run.

\n

In order to do that, as we visit expressions, we need to know if we’re inside\nthe initializer for some variable. We do that by splitting binding into two\nsteps. The first is declaring it.

\n
lox/Resolver.java
\nadd after endScope()
\n
  private void declare(Token name) {\n    if (scopes.isEmpty()) return;\n\n    Map<String, Boolean> scope = scopes.peek();\n    scope.put(name.lexeme, false);\n  }\n
\n
lox/Resolver.java, add after endScope()
\n\n

Declaration adds the variable to the innermost scope so that it shadows any\nouter one and so that we know the variable exists. We mark it as “not ready yet”\nby binding its name to false in the scope map. The value associated with a key\nin the scope map represents whether or not we have finished resolving that\nvariable’s initializer.

\n

After declaring the variable, we resolve its initializer expression in that same\nscope where the new variable now exists but is unavailable. Once the initializer\nexpression is done, the variable is ready for prime time. We do that by\ndefining it.

\n
lox/Resolver.java
\nadd after declare()
\n
  private void define(Token name) {\n    if (scopes.isEmpty()) return;\n    scopes.peek().put(name.lexeme, true);\n  }\n
\n
lox/Resolver.java, add after declare()
\n\n

We set the variable’s value in the scope map to true to mark it as fully\ninitialized and available for use. It’s alive!

\n

11 . 3 . 3Resolving variable expressions

\n

Variable declarations—and function declarations, which we’ll get to—write\nto the scope maps. Those maps are read when we resolve variable expressions.

\n
lox/Resolver.java
\nadd after visitVarStmt()
\n
  @Override\n  public Void visitVariableExpr(Expr.Variable expr) {\n    if (!scopes.isEmpty() &&\n        scopes.peek().get(expr.name.lexeme) == Boolean.FALSE) {\n      Lox.error(expr.name,\n          "Can't read local variable in its own initializer.");\n    }\n\n    resolveLocal(expr, expr.name);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitVarStmt()
\n\n

First, we check to see if the variable is being accessed inside its own\ninitializer. This is where the values in the scope map come into play. If the\nvariable exists in the current scope but its value is false, that means we\nhave declared it but not yet defined it. We report that error.

\n

After that check, we actually resolve the variable itself using this helper:

\n
lox/Resolver.java
\nadd after define()
\n
  private void resolveLocal(Expr expr, Token name) {\n    for (int i = scopes.size() - 1; i >= 0; i--) {\n      if (scopes.get(i).containsKey(name.lexeme)) {\n        interpreter.resolve(expr, scopes.size() - 1 - i);\n        return;\n      }\n    }\n  }\n
\n
lox/Resolver.java, add after define()
\n\n

This looks, for good reason, a lot like the code in Environment for evaluating a\nvariable. We start at the innermost scope and work outwards, looking in each map\nfor a matching name. If we find the variable, we resolve it, passing in the\nnumber of scopes between the current innermost scope and the scope where the\nvariable was found. So, if the variable was found in the current scope, we\npass in 0. If it’s in the immediately enclosing scope, 1. You get the idea.

\n

If we walk through all of the block scopes and never find the variable, we leave\nit unresolved and assume it’s global. We’ll get to the implementation of that\nresolve() method a little later. For now, let’s keep on cranking through the\nother syntax nodes.

\n

11 . 3 . 4Resolving assignment expressions

\n

The other expression that references a variable is assignment. Resolving one\nlooks like this:

\n
lox/Resolver.java
\nadd after visitVarStmt()
\n
  @Override\n  public Void visitAssignExpr(Expr.Assign expr) {\n    resolve(expr.value);\n    resolveLocal(expr, expr.name);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitVarStmt()
\n\n

First, we resolve the expression for the assigned value in case it also contains\nreferences to other variables. Then we use our existing resolveLocal() method\nto resolve the variable that’s being assigned to.

\n

11 . 3 . 5Resolving function declarations

\n

Finally, functions. Functions both bind names and introduce a scope. The name of\nthe function itself is bound in the surrounding scope where the function is\ndeclared. When we step into the function’s body, we also bind its parameters\ninto that inner function scope.

\n
lox/Resolver.java
\nadd after visitBlockStmt()
\n
  @Override\n  public Void visitFunctionStmt(Stmt.Function stmt) {\n    declare(stmt.name);\n    define(stmt.name);\n\n    resolveFunction(stmt);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitBlockStmt()
\n\n

Similar to visitVariableStmt(), we declare and define the name of the function\nin the current scope. Unlike variables, though, we define the name eagerly,\nbefore resolving the function’s body. This lets a function recursively refer to\nitself inside its own body.

\n

Then we resolve the function’s body using this:

\n
lox/Resolver.java
\nadd after resolve()
\n
  private void resolveFunction(Stmt.Function function) {\n    beginScope();\n    for (Token param : function.params) {\n      declare(param);\n      define(param);\n    }\n    resolve(function.body);\n    endScope();\n  }\n
\n
lox/Resolver.java, add after resolve()
\n\n

It’s a separate method since we will also use it for resolving Lox methods when\nwe add classes later. It creates a new scope for the body and then binds\nvariables for each of the function’s parameters.

\n

Once that’s ready, it resolves the function body in that scope. This is\ndifferent from how the interpreter handles function declarations. At runtime,\ndeclaring a function doesn’t do anything with the function’s body. The body\ndoesn’t get touched until later when the function is called. In a static\nanalysis, we immediately traverse into the body right then and there.

\n

11 . 3 . 6Resolving the other syntax tree nodes

\n

That covers the interesting corners of the grammars. We handle every place where\na variable is declared, read, or written, and every place where a scope is\ncreated or destroyed. Even though they aren’t affected by variable resolution,\nwe also need visit methods for all of the other syntax tree nodes in order to\nrecurse into their subtrees. Sorry this bit is\nboring, but bear with me. We’ll go kind of “top down” and start with statements.

\n\n

An expression statement contains a single expression to traverse.

\n
lox/Resolver.java
\nadd after visitBlockStmt()
\n
  @Override\n  public Void visitExpressionStmt(Stmt.Expression stmt) {\n    resolve(stmt.expression);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitBlockStmt()
\n\n

An if statement has an expression for its condition and one or two statements\nfor the branches.

\n
lox/Resolver.java
\nadd after visitFunctionStmt()
\n
  @Override\n  public Void visitIfStmt(Stmt.If stmt) {\n    resolve(stmt.condition);\n    resolve(stmt.thenBranch);\n    if (stmt.elseBranch != null) resolve(stmt.elseBranch);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitFunctionStmt()
\n\n

Here, we see how resolution is different from interpretation. When we resolve an\nif statement, there is no control flow. We resolve the condition and both\nbranches. Where a dynamic execution steps only into the branch that is run, a\nstatic analysis is conservative—it analyzes any branch that could be run.\nSince either one could be reached at runtime, we resolve both.

\n

Like expression statements, a print statement contains a single subexpression.

\n
lox/Resolver.java
\nadd after visitIfStmt()
\n
  @Override\n  public Void visitPrintStmt(Stmt.Print stmt) {\n    resolve(stmt.expression);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitIfStmt()
\n\n

Same deal for return.

\n
lox/Resolver.java
\nadd after visitPrintStmt()
\n
  @Override\n  public Void visitReturnStmt(Stmt.Return stmt) {\n    if (stmt.value != null) {\n      resolve(stmt.value);\n    }\n\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitPrintStmt()
\n\n

As in if statements, with a while statement, we resolve its condition and\nresolve the body exactly once.

\n
lox/Resolver.java
\nadd after visitVarStmt()
\n
  @Override\n  public Void visitWhileStmt(Stmt.While stmt) {\n    resolve(stmt.condition);\n    resolve(stmt.body);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitVarStmt()
\n\n

That covers all the statements. On to expressions . . . 

\n

Our old friend the binary expression. We traverse into and resolve both\noperands.

\n
lox/Resolver.java
\nadd after visitAssignExpr()
\n
  @Override\n  public Void visitBinaryExpr(Expr.Binary expr) {\n    resolve(expr.left);\n    resolve(expr.right);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitAssignExpr()
\n\n

Calls are similar—we walk the argument list and resolve them all. The thing\nbeing called is also an expression (usually a variable expression), so that gets\nresolved too.

\n
lox/Resolver.java
\nadd after visitBinaryExpr()
\n
  @Override\n  public Void visitCallExpr(Expr.Call expr) {\n    resolve(expr.callee);\n\n    for (Expr argument : expr.arguments) {\n      resolve(argument);\n    }\n\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitBinaryExpr()
\n\n

Parentheses are easy.

\n
lox/Resolver.java
\nadd after visitCallExpr()
\n
  @Override\n  public Void visitGroupingExpr(Expr.Grouping expr) {\n    resolve(expr.expression);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitCallExpr()
\n\n

Literals are easiest of all.

\n
lox/Resolver.java
\nadd after visitGroupingExpr()
\n
  @Override\n  public Void visitLiteralExpr(Expr.Literal expr) {\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitGroupingExpr()
\n\n

A literal expression doesn’t mention any variables and doesn’t contain any\nsubexpressions so there is no work to do.

\n

Since a static analysis does no control flow or short-circuiting, logical\nexpressions are exactly the same as other binary operators.

\n
lox/Resolver.java
\nadd after visitLiteralExpr()
\n
  @Override\n  public Void visitLogicalExpr(Expr.Logical expr) {\n    resolve(expr.left);\n    resolve(expr.right);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitLiteralExpr()
\n\n

And, finally, the last node. We resolve its one operand.

\n
lox/Resolver.java
\nadd after visitLogicalExpr()
\n
  @Override\n  public Void visitUnaryExpr(Expr.Unary expr) {\n    resolve(expr.right);\n    return null;\n  }\n
\n
lox/Resolver.java, add after visitLogicalExpr()
\n\n

With all of these visit methods, the Java compiler should be satisfied that\nResolver fully implements Stmt.Visitor and Expr.Visitor. Now is a good time to\ntake a break, have a snack, maybe a little nap.

\n

11 . 4Interpreting Resolved Variables

\n

Let’s see what our resolver is good for. Each time it visits a variable, it\ntells the interpreter how many scopes there are between the current scope and\nthe scope where the variable is defined. At runtime, this corresponds exactly to\nthe number of environments between the current one and the enclosing one where\nthe interpreter can find the variable’s value. The resolver hands that number to\nthe interpreter by calling this:

\n
lox/Interpreter.java
\nadd after execute()
\n
  void resolve(Expr expr, int depth) {\n    locals.put(expr, depth);\n  }\n
\n
lox/Interpreter.java, add after execute()
\n\n

We want to store the resolution information somewhere so we can use it when the\nvariable or assignment expression is later executed, but where? One obvious\nplace is right in the syntax tree node itself. That’s a fine approach, and\nthat’s where many compilers store the results of analyses like this.

\n

We could do that, but it would require mucking around with our syntax tree\ngenerator. Instead, we’ll take another common approach and store it off to the\nside in a map that associates each syntax tree node\nwith its resolved data.

\n\n

Interactive tools like IDEs often incrementally reparse and re-resolve parts of\nthe user’s program. It may be hard to find all of the bits of state that need\nrecalculating when they’re hiding in the foliage of the syntax tree. A benefit\nof storing this data outside of the nodes is that it makes it easy to discard\nit—simply clear the map.

\n
  private Environment environment = globals;\n
lox/Interpreter.java
\nin class Interpreter
\n
  private final Map<Expr, Integer> locals = new HashMap<>();\n
\n\n  Interpreter() {\n
\n
lox/Interpreter.java, in class Interpreter
\n\n

You might think we’d need some sort of nested tree structure to avoid getting\nconfused when there are multiple expressions that reference the same variable,\nbut each expression node is its own Java object with its own unique identity. A\nsingle monolithic map doesn’t have any trouble keeping them separated.

\n

As usual, using a collection requires us to import a couple of names.

\n
import java.util.ArrayList;\n
lox/Interpreter.java
\n
import java.util.HashMap;\n
import java.util.List;\n
\n
lox/Interpreter.java
\n\n

And:

\n
import java.util.List;\n
lox/Interpreter.java
\n
import java.util.Map;\n
\n\nclass Interpreter implements Expr.Visitor<Object>,\n
\n
lox/Interpreter.java
\n\n

11 . 4 . 1Accessing a resolved variable

\n

Our interpreter now has access to each variable’s resolved location. Finally, we\nget to make use of that. We replace the visit method for variable expressions\nwith this:

\n
  public Object visitVariableExpr(Expr.Variable expr) {\n
lox/Interpreter.java
\nin visitVariableExpr()
\nreplace 1 line
\n
    return lookUpVariable(expr.name, expr);\n
  }\n
\n
lox/Interpreter.java, in visitVariableExpr(), replace 1 line
\n\n

That delegates to:

\n
lox/Interpreter.java
\nadd after visitVariableExpr()
\n
  private Object lookUpVariable(Token name, Expr expr) {\n    Integer distance = locals.get(expr);\n    if (distance != null) {\n      return environment.getAt(distance, name.lexeme);\n    } else {\n      return globals.get(name);\n    }\n  }\n
\n
lox/Interpreter.java, add after visitVariableExpr()
\n\n

There are a couple of things going on here. First, we look up the resolved\ndistance in the map. Remember that we resolved only local variables. Globals\nare treated specially and don’t end up in the map (hence the name locals). So,\nif we don’t find a distance in the map, it must be global. In that case, we\nlook it up, dynamically, directly in the global environment. That throws a\nruntime error if the variable isn’t defined.

\n

If we do get a distance, we have a local variable, and we get to take\nadvantage of the results of our static analysis. Instead of calling get(), we\ncall this new method on Environment:

\n
lox/Environment.java
\nadd after define()
\n
  Object getAt(int distance, String name) {\n    return ancestor(distance).values.get(name);\n  }\n
\n
lox/Environment.java, add after define()
\n\n

The old get() method dynamically walks the chain of enclosing environments,\nscouring each one to see if the variable might be hiding in there somewhere. But\nnow we know exactly which environment in the chain will have the variable. We\nreach it using this helper method:

\n
lox/Environment.java
\nadd after define()
\n
  Environment ancestor(int distance) {\n    Environment environment = this;\n    for (int i = 0; i < distance; i++) {\n      environment = environment.enclosing; \n    }\n\n    return environment;\n  }\n
\n
lox/Environment.java, add after define()
\n\n

This walks a fixed number of hops up the parent chain and returns the\nenvironment there. Once we have that, getAt() simply returns the value of the\nvariable in that environment’s map. It doesn’t even have to check to see if the\nvariable is there—we know it will be because the resolver already found it\nbefore.

\n\n

11 . 4 . 2Assigning to a resolved variable

\n

We can also use a variable by assigning to it. The changes to visiting an\nassignment expression are similar.

\n
  public Object visitAssignExpr(Expr.Assign expr) {\n    Object value = evaluate(expr.value);\n
lox/Interpreter.java
\nin visitAssignExpr()
\nreplace 1 line
\n
\n\n    Integer distance = locals.get(expr);\n    if (distance != null) {\n      environment.assignAt(distance, expr.name, value);\n    } else {\n      globals.assign(expr.name, value);\n    }\n\n
    return value;\n
\n
lox/Interpreter.java, in visitAssignExpr(), replace 1 line
\n\n

Again, we look up the variable’s scope distance. If not found, we assume it’s\nglobal and handle it the same way as before. Otherwise, we call this new method:

\n
lox/Environment.java
\nadd after getAt()
\n
  void assignAt(int distance, Token name, Object value) {\n    ancestor(distance).values.put(name.lexeme, value);\n  }\n
\n
lox/Environment.java, add after getAt()
\n\n

As getAt() is to get(), assignAt() is to assign(). It walks a fixed\nnumber of environments, and then stuffs the new value in that map.

\n

Those are the only changes to Interpreter. This is why I chose a representation\nfor our resolved data that was minimally invasive. All of the rest of the nodes\ncontinue working as they did before. Even the code for modifying environments is\nunchanged.

\n

11 . 4 . 3Running the resolver

\n

We do need to actually run the resolver, though. We insert the new pass after\nthe parser does its magic.

\n
    // Stop if there was a syntax error.\n    if (hadError) return;\n\n
lox/Lox.java
\nin run()
\n
    Resolver resolver = new Resolver(interpreter);\n    resolver.resolve(statements);\n\n
    interpreter.interpret(statements);\n
\n
lox/Lox.java, in run()
\n\n

We don’t run the resolver if there are any parse errors. If the code has a\nsyntax error, it’s never going to run, so there’s little value in resolving it.\nIf the syntax is clean, we tell the resolver to do its thing. The resolver has a\nreference to the interpreter and pokes the resolution data directly into it as\nit walks over variables. When the interpreter runs next, it has everything it\nneeds.

\n

At least, that’s true if the resolver succeeds. But what about errors during\nresolution?

\n

11 . 5Resolution Errors

\n

Since we are doing a semantic analysis pass, we have an opportunity to make\nLox’s semantics more precise, and to help users catch bugs early before running\ntheir code. Take a look at this bad boy:

\n
fun bad() {\n  var a = "first";\n  var a = "second";\n}\n
\n

We do allow declaring multiple variables with the same name in the global\nscope, but doing so in a local scope is probably a mistake. If they knew the\nvariable already existed, they would have assigned to it instead of using var.\nAnd if they didn’t know it existed, they probably didn’t intend to overwrite\nthe previous one.

\n

We can detect this mistake statically while resolving.

\n
    Map<String, Boolean> scope = scopes.peek();\n
lox/Resolver.java
\nin declare()
\n
    if (scope.containsKey(name.lexeme)) {\n      Lox.error(name,\n          "Already a variable with this name in this scope.");\n    }\n\n
    scope.put(name.lexeme, false);\n
\n
lox/Resolver.java, in declare()
\n\n

When we declare a variable in a local scope, we already know the names of every\nvariable previously declared in that same scope. If we see a collision, we\nreport an error.

\n

11 . 5 . 1Invalid return errors

\n

Here’s another nasty little script:

\n
return "at top level";\n
\n

This executes a return statement, but it’s not even inside a function at all.\nIt’s top-level code. I don’t know what the user thinks is going to happen, but\nI don’t think we want Lox to allow this.

\n

We can extend the resolver to detect this statically. Much like we track scopes\nas we walk the tree, we can track whether or not the code we are currently\nvisiting is inside a function declaration.

\n
  private final Stack<Map<String, Boolean>> scopes = new Stack<>();\n
lox/Resolver.java
\nin class Resolver
\n
  private FunctionType currentFunction = FunctionType.NONE;\n
\n\n  Resolver(Interpreter interpreter) {\n
\n
lox/Resolver.java, in class Resolver
\n\n

Instead of a bare Boolean, we use this funny enum:

\n
lox/Resolver.java
\nadd after Resolver()
\n
  private enum FunctionType {\n    NONE,\n    FUNCTION\n  }\n
\n
lox/Resolver.java, add after Resolver()
\n\n

It seems kind of dumb now, but we’ll add a couple more cases to it later and\nthen it will make more sense. When we resolve a function declaration, we pass\nthat in.

\n
    define(stmt.name);\n\n
lox/Resolver.java
\nin visitFunctionStmt()
\nreplace 1 line
\n
    resolveFunction(stmt, FunctionType.FUNCTION);\n
    return null;\n
\n
lox/Resolver.java, in visitFunctionStmt(), replace 1 line
\n\n

Over in resolveFunction(), we take that parameter and store it in the field\nbefore resolving the body.

\n
lox/Resolver.java
\nmethod resolveFunction()
\nreplace 1 line
\n
  private void resolveFunction(\n      Stmt.Function function, FunctionType type) {\n    FunctionType enclosingFunction = currentFunction;\n    currentFunction = type;\n\n
    beginScope();\n
\n
lox/Resolver.java, method resolveFunction(), replace 1 line
\n\n

We stash the previous value of the field in a local variable first. Remember,\nLox has local functions, so you can nest function declarations arbitrarily\ndeeply. We need to track not just that we’re in a function, but how many we’re\nin.

\n

We could use an explicit stack of FunctionType values for that, but instead\nwe’ll piggyback on the JVM. We store the previous value in a local on the Java\nstack. When we’re done resolving the function body, we restore the field to that\nvalue.

\n
    endScope();\n
lox/Resolver.java
\nin resolveFunction()
\n
    currentFunction = enclosingFunction;\n
  }\n
\n
lox/Resolver.java, in resolveFunction()
\n\n

Now that we can always tell whether or not we’re inside a function declaration,\nwe check that when resolving a return statement.

\n
  public Void visitReturnStmt(Stmt.Return stmt) {\n
lox/Resolver.java
\nin visitReturnStmt()
\n
    if (currentFunction == FunctionType.NONE) {\n      Lox.error(stmt.keyword, "Can't return from top-level code.");\n    }\n\n
    if (stmt.value != null) {\n
\n
lox/Resolver.java, in visitReturnStmt()
\n\n

Neat, right?

\n

There’s one more piece. Back in the main Lox class that stitches everything\ntogether, we are careful to not run the interpreter if any parse errors are\nencountered. That check runs before the resolver so that we don’t try to\nresolve syntactically invalid code.

\n

But we also need to skip the interpreter if there are resolution errors, so we\nadd another check.

\n
    resolver.resolve(statements);\n
lox/Lox.java
\nin run()
\n
\n\n    // Stop if there was a resolution error.\n    if (hadError) return;\n
\n\n    interpreter.interpret(statements);\n
\n
lox/Lox.java, in run()
\n\n

You could imagine doing lots of other analysis in here. For example, if we added\nbreak statements to Lox, we would probably want to ensure they are only used\ninside loops.

\n

We could go farther and report warnings for code that isn’t necessarily wrong\nbut probably isn’t useful. For example, many IDEs will warn if you have\nunreachable code after a return statement, or a local variable whose value is\nnever read. All of that would be pretty easy to add to our static visiting pass,\nor as separate passes.

\n\n

But, for now, we’ll stick with that limited amount of analysis. The important\npart is that we fixed that one weird annoying edge case bug, though it might be\nsurprising that it took this much work to do it.

\n
\n

Challenges

\n
    \n
  1. \n

    Why is it safe to eagerly define the variable bound to a function’s name\nwhen other variables must wait until after they are initialized before they\ncan be used?

    \n
    2. \n
  2. \n

    How do other languages you know handle local variables that refer to the\nsame name in their initializer, like:

    \n
    var a = "outer";\n{\n  var a = a;\n}\n
    \n

    Is it a runtime error? Compile error? Allowed? Do they treat global\nvariables differently? Do you agree with their choices? Justify your answer.

    \n
    3. \n
  3. \n

    Extend the resolver to report an error if a local variable is never used.

    \n
    4. \n
  4. \n

    Our resolver calculates which environment the variable is found in, but\nit’s still looked up by name in that map. A more efficient environment\nrepresentation would store local variables in an array and look them up by\nindex.

    \n

    Extend the resolver to associate a unique index for each local variable\ndeclared in a scope. When resolving a variable access, look up both the\nscope the variable is in and its index and store that. In the interpreter,\nuse that to quickly access a variable by its index instead of using a map.

    \n
    5. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/scanning-on-demand.html", "content": "\n\n\n\nScanning on Demand · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
16
\n

Scanning on Demand

\n\n
\n

Literature is idiosyncratic arrangements in horizontal lines in only\ntwenty-six phonetic symbols, ten Arabic numbers, and about eight punctuation\nmarks.

\n

Kurt Vonnegut, Like Shaking Hands With God: A Conversation about Writing

\n
\n

Our second interpreter, clox, has three phases—scanner, compiler, and virtual\nmachine. A data structure joins each pair of phases. Tokens flow from scanner to\ncompiler, and chunks of bytecode from compiler to VM. We began our\nimplementation near the end with chunks and the VM. Now, we’re going to\nhop back to the beginning and build a scanner that makes tokens. In the\nnext chapter, we’ll tie the two ends together with our bytecode compiler.

\"Source\n

I’ll admit, this is not the most exciting chapter in the book. With two\nimplementations of the same language, there’s bound to be some redundancy. I did\nsneak in a few interesting differences compared to jlox’s scanner. Read on to\nsee what they are.

\n

16 . 1Spinning Up the Interpreter

\n

Now that we’re building the front end, we can get clox running like a real\ninterpreter. No more hand-authored chunks of bytecode. It’s time for a REPL and\nscript loading. Tear out most of the code in main() and replace it with:

\n
int main(int argc, const char* argv[]) {\n  initVM();\n\n
main.c
\nin main()
\nreplace 26 lines
\n
  if (argc == 1) {\n    repl();\n  } else if (argc == 2) {\n    runFile(argv[1]);\n  } else {\n    fprintf(stderr, "Usage: clox [path]\\n");\n    exit(64);\n  }\n\n  freeVM();\n
  return 0;\n}\n
\n
main.c, in main(), replace 26 lines
\n\n

If you pass no arguments to the executable, you are\ndropped into the REPL. A single command line argument is understood to be the\npath to a script to run.

\n\n

We’ll need a few system headers, so let’s get them all out of the way.

\n
main.c
\nadd to top of file
\n
#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n\n
#include "common.h"\n
\n
main.c, add to top of file
\n\n

Next, we get the REPL up and REPL-ing.

\n
#include "vm.h"\n
main.c
\n
\n\nstatic void repl() {\n  char line[1024];\n  for (;;) {\n    printf("> ");\n\n    if (!fgets(line, sizeof(line), stdin)) {\n      printf("\\n");\n      break;\n    }\n\n    interpret(line);\n  }\n}\n
\n
main.c
\n\n

A quality REPL handles input that spans multiple lines gracefully and doesn’t\nhave a hardcoded line length limit. This REPL here is a little more, ahem,\naustere, but it’s fine for our purposes.

\n

The real work happens in interpret(). We’ll get to that soon, but first let’s\ntake care of loading scripts.

\n
main.c
\nadd after repl()
\n
static void runFile(const char* path) {\n  char* source = readFile(path);\n  InterpretResult result = interpret(source);\n  free(source); \n\n  if (result == INTERPRET_COMPILE_ERROR) exit(65);\n  if (result == INTERPRET_RUNTIME_ERROR) exit(70);\n}\n
\n
main.c, add after repl()
\n\n

We read the file and execute the resulting string of Lox source code. Then,\nbased on the result of that, we set the exit code appropriately because we’re\nscrupulous tool builders and care about little details like that.

\n

We also need to free the source code string because readFile() dynamically\nallocates it and passes ownership to its caller. That function looks like this:

\n\n
main.c
\nadd after repl()
\n
static char* readFile(const char* path) {\n  FILE* file = fopen(path, "rb");\n\n  fseek(file, 0L, SEEK_END);\n  size_t fileSize = ftell(file);\n  rewind(file);\n\n  char* buffer = (char*)malloc(fileSize + 1);\n  size_t bytesRead = fread(buffer, sizeof(char), fileSize, file);\n  buffer[bytesRead] = '\\0';\n\n  fclose(file);\n  return buffer;\n}\n
\n
main.c, add after repl()
\n\n

Like a lot of C code, it takes more effort than it seems like it should,\nespecially for a language expressly designed for operating systems. The\ndifficult part is that we want to allocate a big enough string to read the whole\nfile, but we don’t know how big the file is until we’ve read it.

\n

The code here is the classic trick to solve that. We open the file, but before\nreading it, we seek to the very end using fseek(). Then we call ftell()\nwhich tells us how many bytes we are from the start of the file. Since we seeked\n(sought?) to the end, that’s the size. We rewind back to the beginning, allocate\na string of that size, and read the whole file in a\nsingle batch.

\n\n

So we’re done, right? Not quite. These function calls, like most calls in the C\nstandard library, can fail. If this were Java, the failures would be thrown as\nexceptions and automatically unwind the stack so we wouldn’t really need to\nhandle them. In C, if we don’t check for them, they silently get ignored.

\n

This isn’t really a book on good C programming practice, but I hate to encourage\nbad style, so let’s go ahead and handle the errors. It’s good for us, like\neating our vegetables or flossing.

\n

Fortunately, we don’t need to do anything particularly clever if a failure\noccurs. If we can’t correctly read the user’s script, all we can really do is\ntell the user and exit the interpreter gracefully. First of all, we might fail\nto open the file.

\n
  FILE* file = fopen(path, "rb");\n
main.c
\nin readFile()
\n
  if (file == NULL) {\n    fprintf(stderr, "Could not open file \\"%s\\".\\n", path);\n    exit(74);\n  }\n
\n\n  fseek(file, 0L, SEEK_END);\n
\n
main.c, in readFile()
\n\n

This can happen if the file doesn’t exist or the user doesn’t have access to it.\nIt’s pretty common—people mistype paths all the time.

\n

This failure is much rarer:

\n
  char* buffer = (char*)malloc(fileSize + 1);\n
main.c
\nin readFile()
\n
  if (buffer == NULL) {\n    fprintf(stderr, "Not enough memory to read \\"%s\\".\\n", path);\n    exit(74);\n  }\n\n
  size_t bytesRead = fread(buffer, sizeof(char), fileSize, file);\n
\n
main.c, in readFile()
\n\n

If we can’t even allocate enough memory to read the Lox script, the user’s\nprobably got bigger problems to worry about, but we should do our best to at\nleast let them know.

\n

Finally, the read itself may fail.

\n
  size_t bytesRead = fread(buffer, sizeof(char), fileSize, file);\n
main.c
\nin readFile()
\n
  if (bytesRead < fileSize) {\n    fprintf(stderr, "Could not read file \\"%s\\".\\n", path);\n    exit(74);\n  }\n\n
  buffer[bytesRead] = '\\0';\n
\n
main.c, in readFile()
\n\n

This is also unlikely. Actually, the calls to\nfseek(), ftell(), and rewind() could theoretically fail too, but let’s not\ngo too far off in the weeds, shall we?

\n\n

16 . 1 . 1Opening the compilation pipeline

\n

We’ve got ourselves a string of Lox source code, so now we’re ready to set up a\npipeline to scan, compile, and execute it. It’s driven by interpret(). Right\nnow, that function runs our old hardcoded test chunk. Let’s change it to\nsomething closer to its final incarnation.

\n
void freeVM();\n
vm.h
\nfunction interpret()
\nreplace 1 line
\n
InterpretResult interpret(const char* source);\n
void push(Value value);\n
\n
vm.h, function interpret(), replace 1 line
\n\n

Where before we passed in a Chunk, now we pass in the string of source code.\nHere’s the new implementation:

\n
vm.c
\nfunction interpret()
\nreplace 4 lines
\n
InterpretResult interpret(const char* source) {\n  compile(source);\n  return INTERPRET_OK;\n
}\n
\n
vm.c, function interpret(), replace 4 lines
\n\n

We won’t build the actual compiler yet in this chapter, but we can start\nlaying out its structure. It lives in a new module.

\n
#include "common.h"\n
vm.c
\n
#include "compiler.h"\n
#include "debug.h"\n
\n
vm.c
\n\n

For now, the one function in it is declared like so:

\n
compiler.h
\ncreate new file
\n
#ifndef clox_compiler_h\n#define clox_compiler_h\n\nvoid compile(const char* source);\n\n#endif\n
\n
compiler.h, create new file
\n\n

That signature will change, but it gets us going.

\n

The first phase of compilation is scanning—the thing we’re doing in this\nchapter—so right now all the compiler does is set that up.

\n
compiler.c
\ncreate new file
\n
#include <stdio.h>\n\n#include "common.h"\n#include "compiler.h"\n#include "scanner.h"\n\nvoid compile(const char* source) {\n  initScanner(source);\n}\n
\n
compiler.c, create new file
\n\n

This will also grow in later chapters, naturally.

\n

16 . 1 . 2The scanner scans

\n

There are still a few more feet of scaffolding to stand up before we can start\nwriting useful code. First, a new header:

\n
scanner.h
\ncreate new file
\n
#ifndef clox_scanner_h\n#define clox_scanner_h\n\nvoid initScanner(const char* source);\n\n#endif\n
\n
scanner.h, create new file
\n\n

And its corresponding implementation:

\n
scanner.c
\ncreate new file
\n
#include <stdio.h>\n#include <string.h>\n\n#include "common.h"\n#include "scanner.h"\n\ntypedef struct {\n  const char* start;\n  const char* current;\n  int line;\n} Scanner;\n\nScanner scanner;\n
\n
scanner.c, create new file
\n\n

As our scanner chews through the user’s source code, it tracks how far it’s\ngone. Like we did with the VM, we wrap that state in a struct and then create a\nsingle top-level module variable of that type so we don’t have to pass it around\nall of the various functions.

\n

There are surprisingly few fields. The start pointer marks the beginning of\nthe current lexeme being scanned, and current points to the current character\nbeing looked at.

\n

\"The\n\n

We have a line field to track what line the current lexeme is on for error\nreporting. That’s it! We don’t even keep a pointer to the beginning of the\nsource code string. The scanner works its way through the code once and is done\nafter that.

\n

Since we have some state, we should initialize it.

\n
scanner.c
\nadd after variable scanner
\n
void initScanner(const char* source) {\n  scanner.start = source;\n  scanner.current = source;\n  scanner.line = 1;\n}\n
\n
scanner.c, add after variable scanner
\n\n

We start at the very first character on the very first line, like a runner\ncrouched at the starting line.

\n

16 . 2A Token at a Time

\n

In jlox, when the starting gun went off, the scanner raced ahead and eagerly\nscanned the whole program, returning a list of tokens. This would be a challenge\nin clox. We’d need some sort of growable array or list to store the tokens in.\nWe’d need to manage allocating and freeing the tokens, and the collection\nitself. That’s a lot of code, and a lot of memory churn.

\n

At any point in time, the compiler needs only one or two tokens—remember our\ngrammar requires only a single token of lookahead—so we don’t need to keep\nthem all around at the same time. Instead, the simplest solution is to not\nscan a token until the compiler needs one. When the scanner provides one, it\nreturns the token by value. It doesn’t need to dynamically allocate anything—it can just pass tokens around on the C stack.

\n

Unfortunately, we don’t have a compiler yet that can ask the scanner for tokens,\nso the scanner will just sit there doing nothing. To kick it into action, we’ll\nwrite some temporary code to drive it.

\n
  initScanner(source);\n
compiler.c
\nin compile()
\n
  int line = -1;\n  for (;;) {\n    Token token = scanToken();\n    if (token.line != line) {\n      printf("%4d ", token.line);\n      line = token.line;\n    } else {\n      printf("   | ");\n    }\n    printf("%2d '%.*s'\\n", token.type, token.length, token.start); \n\n    if (token.type == TOKEN_EOF) break;\n  }\n
}\n
\n
compiler.c, in compile()
\n\n\n

This loops indefinitely. Each turn through the loop, it scans one token and\nprints it. When it reaches a special “end of file” token or an error, it stops.\nFor example, if we run the interpreter on this program:

\n
print 1 + 2;\n
\n

It prints out:

\n
   1 31 'print'\n   | 21 '1'\n   |  7 '+'\n   | 21 '2'\n   |  8 ';'\n   2 39 ''\n
\n

The first column is the line number, the second is the numeric value of the\ntoken type, and then finally the lexeme. That last\nempty lexeme on line 2 is the EOF token.

\n\n

The goal for the rest of the chapter is to make that blob of code work by\nimplementing this key function:

\n
void initScanner(const char* source);\n
scanner.h
\nadd after initScanner()
\n
Token scanToken();\n
\n\n#endif\n
\n
scanner.h, add after initScanner()
\n\n

Each call scans and returns the next token in the source code. A token looks\nlike this:

\n
#define clox_scanner_h\n
scanner.h
\n
\n\ntypedef struct {\n  TokenType type;\n  const char* start;\n  int length;\n  int line;\n} Token;\n
\n\nvoid initScanner(const char* source);\n
\n
scanner.h
\n\n

It’s pretty similar to jlox’s Token class. We have an enum identifying what type\nof token it is—number, identifier, + operator, etc. The enum is virtually\nidentical to the one in jlox, so let’s just hammer out the whole thing.

\n
#ifndef clox_scanner_h\n#define clox_scanner_h\n
scanner.h
\n
\n\ntypedef enum {\n  // Single-character tokens.\n  TOKEN_LEFT_PAREN, TOKEN_RIGHT_PAREN,\n  TOKEN_LEFT_BRACE, TOKEN_RIGHT_BRACE,\n  TOKEN_COMMA, TOKEN_DOT, TOKEN_MINUS, TOKEN_PLUS,\n  TOKEN_SEMICOLON, TOKEN_SLASH, TOKEN_STAR,\n  // One or two character tokens.\n  TOKEN_BANG, TOKEN_BANG_EQUAL,\n  TOKEN_EQUAL, TOKEN_EQUAL_EQUAL,\n  TOKEN_GREATER, TOKEN_GREATER_EQUAL,\n  TOKEN_LESS, TOKEN_LESS_EQUAL,\n  // Literals.\n  TOKEN_IDENTIFIER, TOKEN_STRING, TOKEN_NUMBER,\n  // Keywords.\n  TOKEN_AND, TOKEN_CLASS, TOKEN_ELSE, TOKEN_FALSE,\n  TOKEN_FOR, TOKEN_FUN, TOKEN_IF, TOKEN_NIL, TOKEN_OR,\n  TOKEN_PRINT, TOKEN_RETURN, TOKEN_SUPER, TOKEN_THIS,\n  TOKEN_TRUE, TOKEN_VAR, TOKEN_WHILE,\n\n  TOKEN_ERROR, TOKEN_EOF\n} TokenType;\n
\n\ntypedef struct {\n
\n
scanner.h
\n\n

Aside from prefixing all the names with TOKEN_ (since C tosses enum names in\nthe top-level namespace) the only difference is that extra TOKEN_ERROR type.\nWhat’s that about?

\n

There are only a couple of errors that get detected during scanning:\nunterminated strings and unrecognized characters. In jlox, the scanner reports\nthose itself. In clox, the scanner produces a synthetic “error” token for that\nerror and passes it over to the compiler. This way, the compiler knows an error\noccurred and can kick off error recovery before reporting it.

\n

The novel part in clox’s Token type is how it represents the lexeme. In jlox,\neach Token stored the lexeme as its own separate little Java string. If we did\nthat for clox, we’d have to figure out how to manage the memory for those\nstrings. That’s especially hard since we pass tokens by value—multiple tokens could point to the same lexeme string. Ownership gets weird.

\n

Instead, we use the original source string as our character store. We represent\na lexeme by a pointer to its first character and the number of characters it\ncontains. This means we don’t need to worry about managing memory for lexemes at\nall and we can freely copy tokens around. As long as the main source code string\noutlives all of the tokens, everything works fine.

\n\n

16 . 2 . 1Scanning tokens

\n

We’re ready to scan some tokens. We’ll work our way up to the complete\nimplementation, starting with this:

\n
scanner.c
\nadd after initScanner()
\n
Token scanToken() {\n  scanner.start = scanner.current;\n\n  if (isAtEnd()) return makeToken(TOKEN_EOF);\n\n  return errorToken("Unexpected character.");\n}\n
\n
scanner.c, add after initScanner()
\n\n

Since each call to this function scans a complete token, we know we are at the\nbeginning of a new token when we enter the function. Thus, we set\nscanner.start to point to the current character so we remember where the\nlexeme we’re about to scan starts.

\n

Then we check to see if we’ve reached the end of the source code. If so, we\nreturn an EOF token and stop. This is a sentinel value that signals to the\ncompiler to stop asking for more tokens.

\n

If we aren’t at the end, we do some . . . stuff . . . to scan the next token. But we\nhaven’t written that code yet. We’ll get to that soon. If that code doesn’t\nsuccessfully scan and return a token, then we reach the end of the function.\nThat must mean we’re at a character that the scanner can’t recognize, so we\nreturn an error token for that.

\n

This function relies on a couple of helpers, most of which are familiar from\njlox. First up:

\n
scanner.c
\nadd after initScanner()
\n
static bool isAtEnd() {\n  return *scanner.current == '\\0';\n}\n
\n
scanner.c, add after initScanner()
\n\n

We require the source string to be a good null-terminated C string. If the\ncurrent character is the null byte, then we’ve reached the end.

\n

To create a token, we have this constructor-like function:

\n
scanner.c
\nadd after isAtEnd()
\n
static Token makeToken(TokenType type) {\n  Token token;\n  token.type = type;\n  token.start = scanner.start;\n  token.length = (int)(scanner.current - scanner.start);\n  token.line = scanner.line;\n  return token;\n}\n
\n
scanner.c, add after isAtEnd()
\n\n

It uses the scanner’s start and current pointers to capture the token’s\nlexeme. It sets a couple of other obvious fields then returns the token. It has\na sister function for returning error tokens.

\n
scanner.c
\nadd after makeToken()
\n
static Token errorToken(const char* message) {\n  Token token;\n  token.type = TOKEN_ERROR;\n  token.start = message;\n  token.length = (int)strlen(message);\n  token.line = scanner.line;\n  return token;\n}\n
\n
scanner.c, add after makeToken()
\n\n

\n\n

The only difference is that the “lexeme” points to the error message string\ninstead of pointing into the user’s source code. Again, we need to ensure that\nthe error message sticks around long enough for the compiler to read it. In\npractice, we only ever call this function with C string literals. Those are\nconstant and eternal, so we’re fine.

\n

What we have now is basically a working scanner for a language with an empty\nlexical grammar. Since the grammar has no productions, every character is an\nerror. That’s not exactly a fun language to program in, so let’s fill in the\nrules.

\n

16 . 3A Lexical Grammar for Lox

\n

The simplest tokens are only a single character. We recognize those like so:

\n
  if (isAtEnd()) return makeToken(TOKEN_EOF);\n
scanner.c
\nin scanToken()
\n
\n\n  char c = advance();\n\n  switch (c) {\n    case '(': return makeToken(TOKEN_LEFT_PAREN);\n    case ')': return makeToken(TOKEN_RIGHT_PAREN);\n    case '{': return makeToken(TOKEN_LEFT_BRACE);\n    case '}': return makeToken(TOKEN_RIGHT_BRACE);\n    case ';': return makeToken(TOKEN_SEMICOLON);\n    case ',': return makeToken(TOKEN_COMMA);\n    case '.': return makeToken(TOKEN_DOT);\n    case '-': return makeToken(TOKEN_MINUS);\n    case '+': return makeToken(TOKEN_PLUS);\n    case '/': return makeToken(TOKEN_SLASH);\n    case '*': return makeToken(TOKEN_STAR);\n  }\n
\n\n  return errorToken("Unexpected character.");\n
\n
scanner.c, in scanToken()
\n\n

We read the next character from the source code, and then do a straightforward\nswitch to see if it matches any of Lox’s one-character lexemes. To read the next\ncharacter, we use a new helper which consumes the current character and returns\nit.

\n
scanner.c
\nadd after isAtEnd()
\n
static char advance() {\n  scanner.current++;\n  return scanner.current[-1];\n}\n
\n
scanner.c, add after isAtEnd()
\n\n

Next up are the two-character punctuation tokens like != and >=. Each of\nthese also has a corresponding single-character token. That means that when we\nsee a character like !, we don’t know if we’re in a ! token or a != until\nwe look at the next character too. We handle those like so:

\n
    case '*': return makeToken(TOKEN_STAR);\n
scanner.c
\nin scanToken()
\n
    case '!':\n      return makeToken(\n          match('=') ? TOKEN_BANG_EQUAL : TOKEN_BANG);\n    case '=':\n      return makeToken(\n          match('=') ? TOKEN_EQUAL_EQUAL : TOKEN_EQUAL);\n    case '<':\n      return makeToken(\n          match('=') ? TOKEN_LESS_EQUAL : TOKEN_LESS);\n    case '>':\n      return makeToken(\n          match('=') ? TOKEN_GREATER_EQUAL : TOKEN_GREATER);\n
  }\n
\n
scanner.c, in scanToken()
\n\n

After consuming the first character, we look for an =. If found, we consume it\nand return the corresponding two-character token. Otherwise, we leave the\ncurrent character alone (so it can be part of the next token) and return the\nappropriate one-character token.

\n

That logic for conditionally consuming the second character lives here:

\n
scanner.c
\nadd after advance()
\n
static bool match(char expected) {\n  if (isAtEnd()) return false;\n  if (*scanner.current != expected) return false;\n  scanner.current++;\n  return true;\n}\n
\n
scanner.c, add after advance()
\n\n

If the current character is the desired one, we advance and return true.\nOtherwise, we return false to indicate it wasn’t matched.

\n

Now our scanner supports all of the punctuation-like tokens. Before we get to\nthe longer ones, let’s take a little side trip to handle characters that aren’t\npart of a token at all.

\n

16 . 3 . 1Whitespace

\n

Our scanner needs to handle spaces, tabs, and newlines, but those characters\ndon’t become part of any token’s lexeme. We could check for those inside the\nmain character switch in scanToken() but it gets a little tricky to ensure\nthat the function still correctly finds the next token after the whitespace\nwhen you call it. We’d have to wrap the whole body of the function in a loop or\nsomething.

\n

Instead, before starting the token, we shunt off to a separate function.

\n
Token scanToken() {\n
scanner.c
\nin scanToken()
\n
  skipWhitespace();\n
  scanner.start = scanner.current;\n
\n
scanner.c, in scanToken()
\n\n

This advances the scanner past any leading whitespace. After this call returns,\nwe know the very next character is a meaningful one (or we’re at the end of the\nsource code).

\n
scanner.c
\nadd after errorToken()
\n
static void skipWhitespace() {\n  for (;;) {\n    char c = peek();\n    switch (c) {\n      case ' ':\n      case '\\r':\n      case '\\t':\n        advance();\n        break;\n      default:\n        return;\n    }\n  }\n}\n
\n
scanner.c, add after errorToken()
\n\n

It’s sort of a separate mini-scanner. It loops, consuming every whitespace\ncharacter it encounters. We need to be careful that it does not consume any\nnon-whitespace characters. To support that, we use this:

\n
scanner.c
\nadd after advance()
\n
static char peek() {\n  return *scanner.current;\n}\n
\n
scanner.c, add after advance()
\n\n

This simply returns the current character, but doesn’t consume it. The previous\ncode handles all the whitespace characters except for newlines.

\n
        break;\n
scanner.c
\nin skipWhitespace()
\n
      case '\\n':\n        scanner.line++;\n        advance();\n        break;\n
      default:\n        return;\n
\n
scanner.c, in skipWhitespace()
\n\n

When we consume one of those, we also bump the current line number.

\n

16 . 3 . 2Comments

\n

Comments aren’t technically “whitespace”, if you want to get all precise with\nyour terminology, but as far as Lox is concerned, they may as well be, so we\nskip those too.

\n
        break;\n
scanner.c
\nin skipWhitespace()
\n
      case '/':\n        if (peekNext() == '/') {\n          // A comment goes until the end of the line.\n          while (peek() != '\\n' && !isAtEnd()) advance();\n        } else {\n          return;\n        }\n        break;\n
      default:\n        return;\n
\n
scanner.c, in skipWhitespace()
\n\n

Comments start with // in Lox, so as with != and friends, we need a second\ncharacter of lookahead. However, with !=, we still wanted to consume the !\neven if the = wasn’t found. Comments are different. If we don’t find a second\n/, then skipWhitespace() needs to not consume the first slash either.

\n

To handle that, we add:

\n
scanner.c
\nadd after peek()
\n
static char peekNext() {\n  if (isAtEnd()) return '\\0';\n  return scanner.current[1];\n}\n
\n
scanner.c, add after peek()
\n\n

This is like peek() but for one character past the current one. If the current\ncharacter and the next one are both /, we consume them and then any other\ncharacters until the next newline or the end of the source code.

\n

We use peek() to check for the newline but not consume it. That way, the\nnewline will be the current character on the next turn of the outer loop in\nskipWhitespace() and we’ll recognize it and increment scanner.line.

\n

16 . 3 . 3Literal tokens

\n

Number and string tokens are special because they have a runtime value\nassociated with them. We’ll start with strings because they are easy to\nrecognize—they always begin with a double quote.

\n
          match('=') ? TOKEN_GREATER_EQUAL : TOKEN_GREATER);\n
scanner.c
\nin scanToken()
\n
    case '"': return string();\n
  }\n
\n
scanner.c, in scanToken()
\n\n

That calls a new function.

\n
scanner.c
\nadd after skipWhitespace()
\n
static Token string() {\n  while (peek() != '"' && !isAtEnd()) {\n    if (peek() == '\\n') scanner.line++;\n    advance();\n  }\n\n  if (isAtEnd()) return errorToken("Unterminated string.");\n\n  // The closing quote.\n  advance();\n  return makeToken(TOKEN_STRING);\n}\n
\n
scanner.c, add after skipWhitespace()
\n\n

Similar to jlox, we consume characters until we reach the closing quote. We also\ntrack newlines inside the string literal. (Lox supports multi-line strings.)\nAnd, as ever, we gracefully handle running out of source code before we find the\nend quote.

\n

The main change here in clox is something that’s not present. Again, it\nrelates to memory management. In jlox, the Token class had a field of type\nObject to store the runtime value converted from the literal token’s lexeme.

\n

Implementing that in C would require a lot of work. We’d need some sort of union\nand type tag to tell whether the token contains a string or double value. If\nit’s a string, we’d need to manage the memory for the string’s character array\nsomehow.

\n

Instead of adding that complexity to the scanner, we defer converting the literal lexeme to a runtime value until\nlater. In clox, tokens only store the lexeme—the character sequence exactly\nas it appears in the user’s source code. Later in the compiler, we’ll convert\nthat lexeme to a runtime value right when we are ready to store it in the\nchunk’s constant table.

\n\n

Next up, numbers. Instead of adding a switch case for each of the ten digits\nthat can start a number, we handle them here:

\n
  char c = advance();\n
scanner.c
\nin scanToken()
\n
  if (isDigit(c)) return number();\n
\n\n  switch (c) {\n
\n
scanner.c, in scanToken()
\n\n

That uses this obvious utility function:

\n
scanner.c
\nadd after initScanner()
\n
static bool isDigit(char c) {\n  return c >= '0' && c <= '9';\n}\n
\n
scanner.c, add after initScanner()
\n\n

We finish scanning the number using this:

\n
scanner.c
\nadd after skipWhitespace()
\n
static Token number() {\n  while (isDigit(peek())) advance();\n\n  // Look for a fractional part.\n  if (peek() == '.' && isDigit(peekNext())) {\n    // Consume the ".".\n    advance();\n\n    while (isDigit(peek())) advance();\n  }\n\n  return makeToken(TOKEN_NUMBER);\n}\n
\n
scanner.c, add after skipWhitespace()
\n\n

It’s virtually identical to jlox’s version except, again, we don’t convert the\nlexeme to a double yet.

\n

16 . 4Identifiers and Keywords

\n

The last batch of tokens are identifiers, both user-defined and reserved. This\nsection should be fun—the way we recognize keywords in clox is quite\ndifferent from how we did it in jlox, and touches on some important data\nstructures.

\n

First, though, we have to scan the lexeme. Names start with a letter or\nunderscore.

\n
  char c = advance();\n
scanner.c
\nin scanToken()
\n
  if (isAlpha(c)) return identifier();\n
  if (isDigit(c)) return number();\n
\n
scanner.c, in scanToken()
\n\n

We recognize those using this:

\n
scanner.c
\nadd after initScanner()
\n
static bool isAlpha(char c) {\n  return (c >= 'a' && c <= 'z') ||\n         (c >= 'A' && c <= 'Z') ||\n          c == '_';\n}\n
\n
scanner.c, add after initScanner()
\n\n

Once we’ve found an identifier, we scan the rest of it here:

\n
scanner.c
\nadd after skipWhitespace()
\n
static Token identifier() {\n  while (isAlpha(peek()) || isDigit(peek())) advance();\n  return makeToken(identifierType());\n}\n
\n
scanner.c, add after skipWhitespace()
\n\n

After the first letter, we allow digits too, and we keep consuming alphanumerics\nuntil we run out of them. Then we produce a token with the proper type.\nDetermining that “proper” type is the unique part of this chapter.

\n
scanner.c
\nadd after skipWhitespace()
\n
static TokenType identifierType() {\n  return TOKEN_IDENTIFIER;\n}\n
\n
scanner.c, add after skipWhitespace()
\n\n

Okay, I guess that’s not very exciting yet. That’s what it looks like if we\nhave no reserved words at all. How should we go about recognizing keywords? In\njlox, we stuffed them all in a Java Map and looked them up by name. We don’t\nhave any sort of hash table structure in clox, at least not yet.

\n

A hash table would be overkill anyway. To look up a string in a hash table, we need to walk the string to calculate its hash code,\nfind the corresponding bucket in the hash table, and then do a\ncharacter-by-character equality comparison on any string it happens to find\nthere.

\n\n

Let’s say we’ve scanned the identifier “gorgonzola”. How much work should we\nneed to do to tell if that’s a reserved word? Well, no Lox keyword starts with\n“g”, so looking at the first character is enough to definitively answer no.\nThat’s a lot simpler than a hash table lookup.

\n

What about “cardigan”? We do have a keyword in Lox that starts with “c”:\n“class”. But the second character in “cardigan”, “a”, rules that out. What about\n“forest”? Since “for” is a keyword, we have to go farther in the string before\nwe can establish that we don’t have a reserved word. But, in most cases, only a\ncharacter or two is enough to tell we’ve got a user-defined name on our hands.\nWe should be able to recognize that and fail fast.

\n

Here’s a visual representation of that branching character-inspection logic:

\n

\"A\n\n

We start at the root node. If there is a child node whose letter matches the\nfirst character in the lexeme, we move to that node. Then repeat for the next\nletter in the lexeme and so on. If at any point the next letter in the lexeme\ndoesn’t match a child node, then the identifier must not be a keyword and we\nstop. If we reach a double-lined box, and we’re at the last character of the\nlexeme, then we found a keyword.

\n

16 . 4 . 1Tries and state machines

\n

This tree diagram is an example of a thing called a trie. A trie stores a set of strings. Most other\ndata structures for storing strings contain the raw character arrays and then\nwrap them inside some larger construct that helps you search faster. A trie is\ndifferent. Nowhere in the trie will you find a whole string.

\n\n

Instead, each string the trie “contains” is represented as a path through the\ntree of character nodes, as in our traversal above. Nodes that match the last\ncharacter in a string have a special marker—the double lined boxes in the\nillustration. That way, if your trie contains, say, “banquet” and “ban”, you are\nable to tell that it does not contain “banque”—the “e” node won’t have that\nmarker, while the “n” and “t” nodes will.

\n

Tries are a special case of an even more fundamental data structure: a\ndeterministic finite automaton (DFA). You might also know these\nby other names: finite state machine, or just state machine. State\nmachines are rad. They end up useful in everything from game\nprogramming to implementing networking protocols.

\n

In a DFA, you have a set of states with transitions between them, forming a\ngraph. At any point in time, the machine is “in” exactly one state. It gets to\nother states by following transitions. When you use a DFA for lexical analysis,\neach transition is a character that gets matched from the string. Each state\nrepresents a set of allowed characters.

\n

Our keyword tree is exactly a DFA that recognizes Lox keywords. But DFAs are\nmore powerful than simple trees because they can be arbitrary graphs.\nTransitions can form cycles between states. That lets you recognize arbitrarily\nlong strings. For example, here’s a DFA that recognizes number literals:

\n

\"A\n\n

I’ve collapsed the nodes for the ten digits together to keep it more readable,\nbut the basic process works the same—you work through the path, entering\nnodes whenever you consume a corresponding character in the lexeme. If we were\nso inclined, we could construct one big giant DFA that does all of the lexical\nanalysis for Lox, a single state machine that recognizes and spits out all of\nthe tokens we need.

\n

However, crafting that mega-DFA by hand would be\nchallenging. That’s why Lex was created. You give it a simple textual\ndescription of your lexical grammar—a bunch of regular expressions—and it\nautomatically generates a DFA for you and produces a pile of C code that\nimplements it.

\n\n

We won’t go down that road. We already have a perfectly serviceable hand-rolled\nscanner. We just need a tiny trie for recognizing keywords. How should we map\nthat to code?

\n

The absolute simplest solution is to use a switch\nstatement for each node with cases for each branch. We’ll start with the root\nnode and handle the easy keywords.

\n\n
static TokenType identifierType() {\n
scanner.c
\nin identifierType()
\n
  switch (scanner.start[0]) {\n    case 'a': return checkKeyword(1, 2, "nd", TOKEN_AND);\n    case 'c': return checkKeyword(1, 4, "lass", TOKEN_CLASS);\n    case 'e': return checkKeyword(1, 3, "lse", TOKEN_ELSE);\n    case 'i': return checkKeyword(1, 1, "f", TOKEN_IF);\n    case 'n': return checkKeyword(1, 2, "il", TOKEN_NIL);\n    case 'o': return checkKeyword(1, 1, "r", TOKEN_OR);\n    case 'p': return checkKeyword(1, 4, "rint", TOKEN_PRINT);\n    case 'r': return checkKeyword(1, 5, "eturn", TOKEN_RETURN);\n    case 's': return checkKeyword(1, 4, "uper", TOKEN_SUPER);\n    case 'v': return checkKeyword(1, 2, "ar", TOKEN_VAR);\n    case 'w': return checkKeyword(1, 4, "hile", TOKEN_WHILE);\n  }\n\n
  return TOKEN_IDENTIFIER;\n
\n
scanner.c, in identifierType()
\n\n

These are the initial letters that correspond to a single keyword. If we see an\n“s”, the only keyword the identifier could possibly be is super. It might not\nbe, though, so we still need to check the rest of the letters too. In the tree\ndiagram, this is basically that straight path hanging off the “s”.

\n

We won’t roll a switch for each of those nodes. Instead, we have a utility\nfunction that tests the rest of a potential keyword’s lexeme.

\n
scanner.c
\nadd after skipWhitespace()
\n
static TokenType checkKeyword(int start, int length,\n    const char* rest, TokenType type) {\n  if (scanner.current - scanner.start == start + length &&\n      memcmp(scanner.start + start, rest, length) == 0) {\n    return type;\n  }\n\n  return TOKEN_IDENTIFIER;\n}\n
\n
scanner.c, add after skipWhitespace()
\n\n

We use this for all of the unbranching paths in the tree. Once we’ve found a\nprefix that could only be one possible reserved word, we need to verify two\nthings. The lexeme must be exactly as long as the keyword. If the first letter\nis “s”, the lexeme could still be “sup” or “superb”. And the remaining\ncharacters must match exactly—“supar” isn’t good enough.

\n

If we do have the right number of characters, and they’re the ones we want, then\nit’s a keyword, and we return the associated token type. Otherwise, it must be a\nnormal identifier.

\n

We have a couple of keywords where the tree branches again after the first\nletter. If the lexeme starts with “f”, it could be false, for, or fun. So\nwe add another switch for the branches coming off the “f” node.

\n
    case 'e': return checkKeyword(1, 3, "lse", TOKEN_ELSE);\n
scanner.c
\nin identifierType()
\n
    case 'f':\n      if (scanner.current - scanner.start > 1) {\n        switch (scanner.start[1]) {\n          case 'a': return checkKeyword(2, 3, "lse", TOKEN_FALSE);\n          case 'o': return checkKeyword(2, 1, "r", TOKEN_FOR);\n          case 'u': return checkKeyword(2, 1, "n", TOKEN_FUN);\n        }\n      }\n      break;\n
    case 'i': return checkKeyword(1, 1, "f", TOKEN_IF);\n
\n
scanner.c, in identifierType()
\n\n

Before we switch, we need to check that there even is a second letter. “f” by\nitself is a valid identifier too, after all. The other letter that branches is\n“t”.

\n
    case 's': return checkKeyword(1, 4, "uper", TOKEN_SUPER);\n
scanner.c
\nin identifierType()
\n
    case 't':\n      if (scanner.current - scanner.start > 1) {\n        switch (scanner.start[1]) {\n          case 'h': return checkKeyword(2, 2, "is", TOKEN_THIS);\n          case 'r': return checkKeyword(2, 2, "ue", TOKEN_TRUE);\n        }\n      }\n      break;\n
    case 'v': return checkKeyword(1, 2, "ar", TOKEN_VAR);\n
\n
scanner.c, in identifierType()
\n\n

That’s it. A couple of nested switch statements. Not only is this code short, but it’s very, very fast. It does the minimum amount\nof work required to detect a keyword, and bails out as soon as it can tell the\nidentifier will not be a reserved one.

\n

And with that, our scanner is complete.

\n\n
\n

Challenges

\n
    \n
  1. \n

    Many newer languages support string interpolation. Inside a\nstring literal, you have some sort of special delimiters—most commonly\n${ at the beginning and } at the end. Between those delimiters, any\nexpression can appear. When the string literal is executed, the inner\nexpression is evaluated, converted to a string, and then merged with the\nsurrounding string literal.

    \n

    For example, if Lox supported string interpolation, then this . . . 

    \n
    var drink = "Tea";\nvar steep = 4;\nvar cool = 2;\nprint "${drink} will be ready in ${steep + cool} minutes.";\n
    \n

     . . . would print:

    \n
    Tea will be ready in 6 minutes.\n
    \n

    What token types would you define to implement a scanner for string\ninterpolation? What sequence of tokens would you emit for the above string\nliteral?

    \n

    What tokens would you emit for:

    \n
    "Nested ${"interpolation?! Are you ${"mad?!"}"}"\n
    \n

    Consider looking at other language implementations that support\ninterpolation to see how they handle it.

    \n
    2. \n
  2. \n

    Several languages use angle brackets for generics and also have a >> right\nshift operator. This led to a classic problem in early versions of C++:

    \n
    vector<vector<string>> nestedVectors;\n
    \n

    This would produce a compile error because the >> was lexed to a single\nright shift token, not two > tokens. Users were forced to avoid this by\nputting a space between the closing angle brackets.

    \n

    Later versions of C++ are smarter and can handle the above code. Java and C#\nnever had the problem. How do those languages specify and implement this?

    \n
    3. \n
  3. \n

    Many languages, especially later in their evolution, define “contextual\nkeywords”. These are identifiers that act like reserved words in some\ncontexts but can be normal user-defined identifiers in others.

    \n

    For example, await is a keyword inside an async method in C#, but\nin other methods, you can use await as your own identifier.

    \n

    Name a few contextual keywords from other languages, and the context where\nthey are meaningful. What are the pros and cons of having contextual\nkeywords? How would you implement them in your language’s front end if you\nneeded to?

    \n
    4. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/scanning.html", "content": "\n\n\n\nScanning · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
4
\n

Scanning

\n\n
\n

Take big bites. Anything worth doing is worth overdoing.

\n

Robert A. Heinlein, Time Enough for Love

\n
\n

The first step in any compiler or interpreter is scanning. The scanner takes in raw source code as a series\nof characters and groups it into a series of chunks we call tokens. These\nare the meaningful “words” and “punctuation” that make up the language’s\ngrammar.

\n\n

Scanning is a good starting point for us too because the code isn’t very hard—pretty much a switch statement with delusions of grandeur. It will help us\nwarm up before we tackle some of the more interesting material later. By the end\nof this chapter, we’ll have a full-featured, fast scanner that can take any\nstring of Lox source code and produce the tokens that we’ll feed into the parser\nin the next chapter.

\n

4 . 1The Interpreter Framework

\n

Since this is our first real chapter, before we get to actually scanning some\ncode we need to sketch out the basic shape of our interpreter, jlox. Everything\nstarts with a class in Java.

\n
lox/Lox.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.io.BufferedReader;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.nio.charset.Charset;\nimport java.nio.file.Files;\nimport java.nio.file.Paths;\nimport java.util.List;\n\npublic class Lox {\n  public static void main(String[] args) throws IOException {\n    if (args.length > 1) {\n      System.out.println("Usage: jlox [script]");\n      System.exit(64); \n    } else if (args.length == 1) {\n      runFile(args[0]);\n    } else {\n      runPrompt();\n    }\n  }\n}\n
\n
lox/Lox.java, create new file
\n\n\n

Stick that in a text file, and go get your IDE or Makefile or whatever set up.\nI’ll be right here when you’re ready. Good? OK!

\n

Lox is a scripting language, which means it executes directly from source. Our\ninterpreter supports two ways of running code. If you start jlox from the\ncommand line and give it a path to a file, it reads the file and executes it.

\n
lox/Lox.java
\nadd after main()
\n
  private static void runFile(String path) throws IOException {\n    byte[] bytes = Files.readAllBytes(Paths.get(path));\n    run(new String(bytes, Charset.defaultCharset()));\n  }\n
\n
lox/Lox.java, add after main()
\n\n

If you want a more intimate conversation with your interpreter, you can also run\nit interactively. Fire up jlox without any arguments, and it drops you into a\nprompt where you can enter and execute code one line at a time.

\n\n
lox/Lox.java
\nadd after runFile()
\n
  private static void runPrompt() throws IOException {\n    InputStreamReader input = new InputStreamReader(System.in);\n    BufferedReader reader = new BufferedReader(input);\n\n    for (;;) { \n      System.out.print("> ");\n      String line = reader.readLine();\n      if (line == null) break;\n      run(line);\n    }\n  }\n
\n
lox/Lox.java, add after runFile()
\n\n

The readLine() function, as the name so helpfully implies, reads a line of\ninput from the user on the command line and returns the result. To kill an\ninteractive command-line app, you usually type Control-D. Doing so signals an\n“end-of-file” condition to the program. When that happens readLine() returns\nnull, so we check for that to exit the loop.

\n

Both the prompt and the file runner are thin wrappers around this core function:

\n
lox/Lox.java
\nadd after runPrompt()
\n
  private static void run(String source) {\n    Scanner scanner = new Scanner(source);\n    List<Token> tokens = scanner.scanTokens();\n\n    // For now, just print the tokens.\n    for (Token token : tokens) {\n      System.out.println(token);\n    }\n  }\n
\n
lox/Lox.java, add after runPrompt()
\n\n

It’s not super useful yet since we haven’t written the interpreter, but baby\nsteps, you know? Right now, it prints out the tokens our forthcoming scanner\nwill emit so that we can see if we’re making progress.

\n

4 . 1 . 1Error handling

\n

While we’re setting things up, another key piece of infrastructure is error\nhandling. Textbooks sometimes gloss over this because it’s more a practical\nmatter than a formal computer science-y problem. But if you care about making a\nlanguage that’s actually usable, then handling errors gracefully is vital.

\n

The tools our language provides for dealing with errors make up a large portion\nof its user interface. When the user’s code is working, they aren’t thinking\nabout our language at all—their headspace is all about their program. It’s\nusually only when things go wrong that they notice our implementation.

\n

When that happens, it’s up to us to give the user all\nthe information they need to understand what went wrong and guide them gently\nback to where they are trying to go. Doing that well means thinking about error\nhandling all through the implementation of our interpreter, starting now.

\n\n
lox/Lox.java
\nadd after run()
\n
  static void error(int line, String message) {\n    report(line, "", message);\n  }\n\n  private static void report(int line, String where,\n                             String message) {\n    System.err.println(\n        "[line " + line + "] Error" + where + ": " + message);\n    hadError = true;\n  }\n
\n
lox/Lox.java, add after run()
\n\n

This error() function and its report() helper tells the user some syntax\nerror occurred on a given line. That is really the bare minimum to be able to\nclaim you even have error reporting. Imagine if you accidentally left a\ndangling comma in some function call and the interpreter printed out:

\n
Error: Unexpected "," somewhere in your code. Good luck finding it!\n
\n

That’s not very helpful. We need to at least point them to the right line. Even\nbetter would be the beginning and end column so they know where in the line.\nEven better than that is to show the user the offending line, like:

\n
Error: Unexpected "," in argument list.\n\n    15 | function(first, second,);\n                               ^-- Here.\n
\n

I’d love to implement something like that in this book but the honest truth is\nthat it’s a lot of grungy string manipulation code. Very useful for users, but\nnot super fun to read in a book and not very technically interesting. So we’ll\nstick with just a line number. In your own interpreters, please do as I say and\nnot as I do.

\n

The primary reason we’re sticking this error reporting function in the main Lox\nclass is because of that hadError field. It’s defined here:

\n
public class Lox {\n
lox/Lox.java
\nin class Lox
\n
  static boolean hadError = false;\n
\n
lox/Lox.java, in class Lox
\n\n

We’ll use this to ensure we don’t try to execute code that has a known error.\nAlso, it lets us exit with a non-zero exit code like a good command line citizen\nshould.

\n
    run(new String(bytes, Charset.defaultCharset()));\n
lox/Lox.java
\nin runFile()
\n
\n\n    // Indicate an error in the exit code.\n    if (hadError) System.exit(65);\n
  }\n
\n
lox/Lox.java, in runFile()
\n\n

We need to reset this flag in the interactive loop. If the user makes a mistake,\nit shouldn’t kill their entire session.

\n
      run(line);\n
lox/Lox.java
\nin runPrompt()
\n
      hadError = false;\n
    }\n
\n
lox/Lox.java, in runPrompt()
\n\n

The other reason I pulled the error reporting out here instead of stuffing it\ninto the scanner and other phases where the error might occur is to remind you\nthat it’s good engineering practice to separate the code that generates the\nerrors from the code that reports them.

\n

Various phases of the front end will detect errors, but it’s not really their\njob to know how to present that to a user. In a full-featured language\nimplementation, you will likely have multiple ways errors get displayed: on\nstderr, in an IDE’s error window, logged to a file, etc. You don’t want that\ncode smeared all over your scanner and parser.

\n

Ideally, we would have an actual abstraction, some kind of “ErrorReporter” interface that gets passed to the scanner\nand parser so that we can swap out different reporting strategies. For our\nsimple interpreter here, I didn’t do that, but I did at least move the code for\nerror reporting into a different class.

\n\n

With some rudimentary error handling in place, our application shell is ready.\nOnce we have a Scanner class with a scanTokens() method, we can start running\nit. Before we get to that, let’s get more precise about what tokens are.

\n

4 . 2Lexemes and Tokens

\n

Here’s a line of Lox code:

\n
var language = "lox";\n
\n

Here, var is the keyword for declaring a variable. That three-character\nsequence “v-a-r” means something. But if we yank three letters out of the\nmiddle of language, like “g-u-a”, those don’t mean anything on their own.

\n

That’s what lexical analysis is about. Our job is to scan through the list of\ncharacters and group them together into the smallest sequences that still\nrepresent something. Each of these blobs of characters is called a lexeme.\nIn that example line of code, the lexemes are:

\"'var',\n

The lexemes are only the raw substrings of the source code. However, in the\nprocess of grouping character sequences into lexemes, we also stumble upon some\nother useful information. When we take the lexeme and bundle it together with\nthat other data, the result is a token. It includes useful stuff like:

\n

4 . 2 . 1Token type

\n

Keywords are part of the shape of the language’s grammar, so the parser often\nhas code like, “If the next token is while then do . . . ” That means the parser\nwants to know not just that it has a lexeme for some identifier, but that it has\na reserved word, and which keyword it is.

\n

The parser could categorize tokens from the raw lexeme\nby comparing the strings, but that’s slow and kind of ugly. Instead, at the\npoint that we recognize a lexeme, we also remember which kind of lexeme it\nrepresents. We have a different type for each keyword, operator, bit of\npunctuation, and literal type.

\n\n
lox/TokenType.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nenum TokenType {\n  // Single-character tokens.\n  LEFT_PAREN, RIGHT_PAREN, LEFT_BRACE, RIGHT_BRACE,\n  COMMA, DOT, MINUS, PLUS, SEMICOLON, SLASH, STAR,\n\n  // One or two character tokens.\n  BANG, BANG_EQUAL,\n  EQUAL, EQUAL_EQUAL,\n  GREATER, GREATER_EQUAL,\n  LESS, LESS_EQUAL,\n\n  // Literals.\n  IDENTIFIER, STRING, NUMBER,\n\n  // Keywords.\n  AND, CLASS, ELSE, FALSE, FUN, FOR, IF, NIL, OR,\n  PRINT, RETURN, SUPER, THIS, TRUE, VAR, WHILE,\n\n  EOF\n}\n
\n
lox/TokenType.java, create new file
\n\n

4 . 2 . 2Literal value

\n

There are lexemes for literal values—numbers and strings and the like. Since\nthe scanner has to walk each character in the literal to correctly identify it,\nit can also convert that textual representation of a value to the living runtime\nobject that will be used by the interpreter later.

\n

4 . 2 . 3Location information

\n

Back when I was preaching the gospel about error handling, we saw that we need\nto tell users where errors occurred. Tracking that starts here. In our simple\ninterpreter, we note only which line the token appears on, but more\nsophisticated implementations include the column and length too.

\n\n

We take all of this data and wrap it in a class.

\n
lox/Token.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nclass Token {\n  final TokenType type;\n  final String lexeme;\n  final Object literal;\n  final int line; \n\n  Token(TokenType type, String lexeme, Object literal, int line) {\n    this.type = type;\n    this.lexeme = lexeme;\n    this.literal = literal;\n    this.line = line;\n  }\n\n  public String toString() {\n    return type + " " + lexeme + " " + literal;\n  }\n}\n
\n
lox/Token.java, create new file
\n\n

Now we have an object with enough structure to be useful for all of the later\nphases of the interpreter.

\n

4 . 3Regular Languages and Expressions

\n

Now that we know what we’re trying to produce, let’s, well, produce it. The core\nof the scanner is a loop. Starting at the first character of the source code,\nthe scanner figures out what lexeme the character belongs to, and consumes it\nand any following characters that are part of that lexeme. When it reaches the\nend of that lexeme, it emits a token.

\n

Then it loops back and does it again, starting from the very next character in\nthe source code. It keeps doing that, eating characters and occasionally, uh,\nexcreting tokens, until it reaches the end of the input.

\n

\"An\n\n

The part of the loop where we look at a handful of characters to figure out\nwhich kind of lexeme it “matches” may sound familiar. If you know regular\nexpressions, you might consider defining a regex for each kind of lexeme and\nusing those to match characters. For example, Lox has the same rules as C for\nidentifiers (variable names and the like). This regex matches one:

\n
[a-zA-Z_][a-zA-Z_0-9]*\n
\n

If you did think of regular expressions, your intuition is a deep one. The rules\nthat determine how a particular language groups characters into lexemes are\ncalled its lexical grammar. In Lox, as in most\nprogramming languages, the rules of that grammar are simple enough for the\nlanguage to be classified a regular language. That’s the same “regular”\nas in regular expressions.

\n\n

You very precisely can recognize all of the different lexemes for Lox using\nregexes if you want to, and there’s a pile of interesting theory underlying why\nthat is and what it means. Tools like Lex or\nFlex are designed expressly to let you do this—throw a handful of regexes\nat them, and they give you a complete scanner back.

\n\n

Since our goal is to understand how a scanner does what it does, we won’t be\ndelegating that task. We’re about handcrafted goods.

\n

4 . 4The Scanner Class

\n

Without further ado, let’s make ourselves a scanner.

\n
lox/Scanner.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\n\nimport static com.craftinginterpreters.lox.TokenType.*; \n\nclass Scanner {\n  private final String source;\n  private final List<Token> tokens = new ArrayList<>();\n\n  Scanner(String source) {\n    this.source = source;\n  }\n}\n
\n
lox/Scanner.java, create new file
\n\n\n

We store the raw source code as a simple string, and we have a list ready to\nfill with tokens we’re going to generate. The aforementioned loop that does that\nlooks like this:

\n
lox/Scanner.java
\nadd after Scanner()
\n
  List<Token> scanTokens() {\n    while (!isAtEnd()) {\n      // We are at the beginning of the next lexeme.\n      start = current;\n      scanToken();\n    }\n\n    tokens.add(new Token(EOF, "", null, line));\n    return tokens;\n  }\n
\n
lox/Scanner.java, add after Scanner()
\n\n

The scanner works its way through the source code, adding tokens until it runs\nout of characters. Then it appends one final “end of file” token. That isn’t\nstrictly needed, but it makes our parser a little cleaner.

\n

This loop depends on a couple of fields to keep track of where the scanner is in\nthe source code.

\n
  private final List<Token> tokens = new ArrayList<>();\n
lox/Scanner.java
\nin class Scanner
\n
  private int start = 0;\n  private int current = 0;\n  private int line = 1;\n
\n\n  Scanner(String source) {\n
\n
lox/Scanner.java, in class Scanner
\n\n

The start and current fields are offsets that index into the string. The\nstart field points to the first character in the lexeme being scanned, and\ncurrent points at the character currently being considered. The line field\ntracks what source line current is on so we can produce tokens that know their\nlocation.

\n

Then we have one little helper function that tells us if we’ve consumed all the\ncharacters.

\n
lox/Scanner.java
\nadd after scanTokens()
\n
  private boolean isAtEnd() {\n    return current >= source.length();\n  }\n
\n
lox/Scanner.java, add after scanTokens()
\n\n

4 . 5Recognizing Lexemes

\n

In each turn of the loop, we scan a single token. This is the real heart of the\nscanner. We’ll start simple. Imagine if every lexeme were only a single character\nlong. All you would need to do is consume the next character and pick a token type for\nit. Several lexemes are only a single character in Lox, so let’s start with\nthose.

\n
lox/Scanner.java
\nadd after scanTokens()
\n
  private void scanToken() {\n    char c = advance();\n    switch (c) {\n      case '(': addToken(LEFT_PAREN); break;\n      case ')': addToken(RIGHT_PAREN); break;\n      case '{': addToken(LEFT_BRACE); break;\n      case '}': addToken(RIGHT_BRACE); break;\n      case ',': addToken(COMMA); break;\n      case '.': addToken(DOT); break;\n      case '-': addToken(MINUS); break;\n      case '+': addToken(PLUS); break;\n      case ';': addToken(SEMICOLON); break;\n      case '*': addToken(STAR); break; \n    }\n  }\n
\n
lox/Scanner.java, add after scanTokens()
\n\n\n

Again, we need a couple of helper methods.

\n
lox/Scanner.java
\nadd after isAtEnd()
\n
  private char advance() {\n    return source.charAt(current++);\n  }\n\n  private void addToken(TokenType type) {\n    addToken(type, null);\n  }\n\n  private void addToken(TokenType type, Object literal) {\n    String text = source.substring(start, current);\n    tokens.add(new Token(type, text, literal, line));\n  }\n
\n
lox/Scanner.java, add after isAtEnd()
\n\n

The advance() method consumes the next character in the source file and\nreturns it. Where advance() is for input, addToken() is for output. It grabs\nthe text of the current lexeme and creates a new token for it. We’ll use the\nother overload to handle tokens with literal values soon.

\n

4 . 5 . 1Lexical errors

\n

Before we get too far in, let’s take a moment to think about errors at the\nlexical level. What happens if a user throws a source file containing some\ncharacters Lox doesn’t use, like @#^, at our interpreter? Right now, those\ncharacters get silently discarded. They aren’t used by the Lox language, but\nthat doesn’t mean the interpreter can pretend they aren’t there. Instead, we\nreport an error.

\n
      case '*': addToken(STAR); break; \n
lox/Scanner.java
\nin scanToken()
\n
\n\n      default:\n        Lox.error(line, "Unexpected character.");\n        break;\n
    }\n
\n
lox/Scanner.java, in scanToken()
\n\n

Note that the erroneous character is still consumed by the earlier call to\nadvance(). That’s important so that we don’t get stuck in an infinite loop.

\n

Note also that we keep scanning. There may be\nother errors later in the program. It gives our users a better experience if we\ndetect as many of those as possible in one go. Otherwise, they see one tiny\nerror and fix it, only to have the next error appear, and so on. Syntax error\nWhac-A-Mole is no fun.

\n

(Don’t worry. Since hadError gets set, we’ll never try to execute any of the\ncode, even though we keep going and scan the rest of it.)

\n\n

4 . 5 . 2Operators

\n

We have single-character lexemes working, but that doesn’t cover all of Lox’s\noperators. What about !? It’s a single character, right? Sometimes, yes, but\nif the very next character is an equals sign, then we should instead create a\n!= lexeme. Note that the ! and = are not two independent operators. You\ncan’t write ! = in Lox and have it behave like an inequality operator.\nThat’s why we need to scan != as a single lexeme. Likewise, <, >, and =\ncan all be followed by = to create the other equality and comparison\noperators.

\n

For all of these, we need to look at the second character.

\n
      case '*': addToken(STAR); break; \n
lox/Scanner.java
\nin scanToken()
\n
      case '!':\n        addToken(match('=') ? BANG_EQUAL : BANG);\n        break;\n      case '=':\n        addToken(match('=') ? EQUAL_EQUAL : EQUAL);\n        break;\n      case '<':\n        addToken(match('=') ? LESS_EQUAL : LESS);\n        break;\n      case '>':\n        addToken(match('=') ? GREATER_EQUAL : GREATER);\n        break;\n
\n\n      default:\n
\n
lox/Scanner.java, in scanToken()
\n\n

Those cases use this new method:

\n
lox/Scanner.java
\nadd after scanToken()
\n
  private boolean match(char expected) {\n    if (isAtEnd()) return false;\n    if (source.charAt(current) != expected) return false;\n\n    current++;\n    return true;\n  }\n
\n
lox/Scanner.java, add after scanToken()
\n\n

It’s like a conditional advance(). We only consume the current character if\nit’s what we’re looking for.

\n

Using match(), we recognize these lexemes in two stages. When we reach, for\nexample, !, we jump to its switch case. That means we know the lexeme starts\nwith !. Then we look at the next character to determine if we’re on a != or\nmerely a !.

\n

4 . 6Longer Lexemes

\n

We’re still missing one operator: / for division. That character needs a\nlittle special handling because comments begin with a slash too.

\n
        break;\n
lox/Scanner.java
\nin scanToken()
\n
      case '/':\n        if (match('/')) {\n          // A comment goes until the end of the line.\n          while (peek() != '\\n' && !isAtEnd()) advance();\n        } else {\n          addToken(SLASH);\n        }\n        break;\n
\n\n      default:\n
\n
lox/Scanner.java, in scanToken()
\n\n

This is similar to the other two-character operators, except that when we find a\nsecond /, we don’t end the token yet. Instead, we keep consuming characters\nuntil we reach the end of the line.

\n

This is our general strategy for handling longer lexemes. After we detect the\nbeginning of one, we shunt over to some lexeme-specific code that keeps eating\ncharacters until it sees the end.

\n

We’ve got another helper:

\n
lox/Scanner.java
\nadd after match()
\n
  private char peek() {\n    if (isAtEnd()) return '\\0';\n    return source.charAt(current);\n  }\n
\n
lox/Scanner.java, add after match()
\n\n

It’s sort of like advance(), but doesn’t consume the character. This is called\nlookahead. Since it only looks at the current\nunconsumed character, we have one character of lookahead. The smaller this\nnumber is, generally, the faster the scanner runs. The rules of the lexical\ngrammar dictate how much lookahead we need. Fortunately, most languages in wide\nuse peek only one or two characters ahead.

\n\n

Comments are lexemes, but they aren’t meaningful, and the parser doesn’t want\nto deal with them. So when we reach the end of the comment, we don’t call\naddToken(). When we loop back around to start the next lexeme, start gets\nreset and the comment’s lexeme disappears in a puff of smoke.

\n

While we’re at it, now’s a good time to skip over those other meaningless\ncharacters: newlines and whitespace.

\n
        break;\n
lox/Scanner.java
\nin scanToken()
\n
\n\n      case ' ':\n      case '\\r':\n      case '\\t':\n        // Ignore whitespace.\n        break;\n\n      case '\\n':\n        line++;\n        break;\n
\n\n      default:\n        Lox.error(line, "Unexpected character.");\n
\n
lox/Scanner.java, in scanToken()
\n\n

When encountering whitespace, we simply go back to the beginning of the scan\nloop. That starts a new lexeme after the whitespace character. For newlines,\nwe do the same thing, but we also increment the line counter. (This is why we\nused peek() to find the newline ending a comment instead of match(). We want\nthat newline to get us here so we can update line.)

\n

Our scanner is getting smarter. It can handle fairly free-form code like:

\n
// this is a comment\n(( )){} // grouping stuff\n!*+-/=<> <= == // operators\n
\n

4 . 6 . 1String literals

\n

Now that we’re comfortable with longer lexemes, we’re ready to tackle literals.\nWe’ll do strings first, since they always begin with a specific character, \".

\n
        break;\n
lox/Scanner.java
\nin scanToken()
\n
\n\n      case '"': string(); break;\n
\n\n      default:\n
\n
lox/Scanner.java, in scanToken()
\n\n

That calls:

\n
lox/Scanner.java
\nadd after scanToken()
\n
  private void string() {\n    while (peek() != '"' && !isAtEnd()) {\n      if (peek() == '\\n') line++;\n      advance();\n    }\n\n    if (isAtEnd()) {\n      Lox.error(line, "Unterminated string.");\n      return;\n    }\n\n    // The closing ".\n    advance();\n\n    // Trim the surrounding quotes.\n    String value = source.substring(start + 1, current - 1);\n    addToken(STRING, value);\n  }\n
\n
lox/Scanner.java, add after scanToken()
\n\n

Like with comments, we consume characters until we hit the \" that ends the\nstring. We also gracefully handle running out of input before the string is\nclosed and report an error for that.

\n

For no particular reason, Lox supports multi-line strings. There are pros and\ncons to that, but prohibiting them was a little more complex than allowing them,\nso I left them in. That does mean we also need to update line when we hit a\nnewline inside a string.

\n

Finally, the last interesting bit is that when we create the token, we also\nproduce the actual string value that will be used later by the interpreter.\nHere, that conversion only requires a substring() to strip off the surrounding\nquotes. If Lox supported escape sequences like \\n, we’d unescape those here.

\n

4 . 6 . 2Number literals

\n

All numbers in Lox are floating point at runtime, but both integer and decimal\nliterals are supported. A number literal is a series of digits optionally followed by a . and one or more trailing\ndigits.

\n\n
1234\n12.34\n
\n

We don’t allow a leading or trailing decimal point, so these are both invalid:

\n
.1234\n1234.\n
\n

We could easily support the former, but I left it out to keep things simple. The\nlatter gets weird if we ever want to allow methods on numbers like 123.sqrt().

\n

To recognize the beginning of a number lexeme, we look for any digit. It’s kind\nof tedious to add cases for every decimal digit, so we’ll stuff it in the\ndefault case instead.

\n
      default:\n
lox/Scanner.java
\nin scanToken()
\nreplace 1 line
\n
        if (isDigit(c)) {\n          number();\n        } else {\n          Lox.error(line, "Unexpected character.");\n        }\n
        break;\n
\n
lox/Scanner.java, in scanToken(), replace 1 line
\n\n

This relies on this little utility:

\n
lox/Scanner.java
\nadd after peek()
\n
  private boolean isDigit(char c) {\n    return c >= '0' && c <= '9';\n  } \n
\n
lox/Scanner.java, add after peek()
\n\n\n

Once we know we are in a number, we branch to a separate method to consume the\nrest of the literal, like we do with strings.

\n
lox/Scanner.java
\nadd after scanToken()
\n
  private void number() {\n    while (isDigit(peek())) advance();\n\n    // Look for a fractional part.\n    if (peek() == '.' && isDigit(peekNext())) {\n      // Consume the "."\n      advance();\n\n      while (isDigit(peek())) advance();\n    }\n\n    addToken(NUMBER,\n        Double.parseDouble(source.substring(start, current)));\n  }\n
\n
lox/Scanner.java, add after scanToken()
\n\n

We consume as many digits as we find for the integer part of the literal. Then\nwe look for a fractional part, which is a decimal point (.) followed by at\nleast one digit. If we do have a fractional part, again, we consume as many\ndigits as we can find.

\n

Looking past the decimal point requires a second character of lookahead since we\ndon’t want to consume the . until we’re sure there is a digit after it. So\nwe add:

\n
lox/Scanner.java
\nadd after peek()
\n
  private char peekNext() {\n    if (current + 1 >= source.length()) return '\\0';\n    return source.charAt(current + 1);\n  } \n
\n
lox/Scanner.java, add after peek()
\n\n\n

Finally, we convert the lexeme to its numeric value. Our interpreter uses Java’s\nDouble type to represent numbers, so we produce a value of that type. We’re\nusing Java’s own parsing method to convert the lexeme to a real Java double. We\ncould implement that ourselves, but, honestly, unless you’re trying to cram for\nan upcoming programming interview, it’s not worth your time.

\n

The remaining literals are Booleans and nil, but we handle those as keywords,\nwhich gets us to . . . 

\n

4 . 7Reserved Words and Identifiers

\n

Our scanner is almost done. The only remaining pieces of the lexical grammar to\nimplement are identifiers and their close cousins, the reserved words. You might\nthink we could match keywords like or in the same way we handle\nmultiple-character operators like <=.

\n
case 'o':\n  if (match('r')) {\n    addToken(OR);\n  }\n  break;\n
\n

Consider what would happen if a user named a variable orchid. The scanner\nwould see the first two letters, or, and immediately emit an or keyword\ntoken. This gets us to an important principle called maximal munch. When two lexical grammar rules can both\nmatch a chunk of code that the scanner is looking at, whichever one matches the\nmost characters wins.

\n

That rule states that if we can match orchid as an identifier and or as a\nkeyword, then the former wins. This is also why we tacitly assumed, previously,\nthat <= should be scanned as a single <= token and not < followed by =.

\n\n

Maximal munch means we can’t easily detect a reserved word until we’ve reached\nthe end of what might instead be an identifier. After all, a reserved word is\nan identifier, it’s just one that has been claimed by the language for its own\nuse. That’s where the term reserved word comes from.

\n

So we begin by assuming any lexeme starting with a letter or underscore is an\nidentifier.

\n
      default:\n        if (isDigit(c)) {\n          number();\n
lox/Scanner.java
\nin scanToken()
\n
        } else if (isAlpha(c)) {\n          identifier();\n
        } else {\n          Lox.error(line, "Unexpected character.");\n        }\n
\n
lox/Scanner.java, in scanToken()
\n\n

The rest of the code lives over here:

\n
lox/Scanner.java
\nadd after scanToken()
\n
  private void identifier() {\n    while (isAlphaNumeric(peek())) advance();\n\n    addToken(IDENTIFIER);\n  }\n
\n
lox/Scanner.java, add after scanToken()
\n\n

We define that in terms of these helpers:

\n
lox/Scanner.java
\nadd after peekNext()
\n
  private boolean isAlpha(char c) {\n    return (c >= 'a' && c <= 'z') ||\n           (c >= 'A' && c <= 'Z') ||\n            c == '_';\n  }\n\n  private boolean isAlphaNumeric(char c) {\n    return isAlpha(c) || isDigit(c);\n  }\n
\n
lox/Scanner.java, add after peekNext()
\n\n

That gets identifiers working. To handle keywords, we see if the identifier’s\nlexeme is one of the reserved words. If so, we use a token type specific to that\nkeyword. We define the set of reserved words in a map.

\n
lox/Scanner.java
\nin class Scanner
\n
  private static final Map<String, TokenType> keywords;\n\n  static {\n    keywords = new HashMap<>();\n    keywords.put("and",    AND);\n    keywords.put("class",  CLASS);\n    keywords.put("else",   ELSE);\n    keywords.put("false",  FALSE);\n    keywords.put("for",    FOR);\n    keywords.put("fun",    FUN);\n    keywords.put("if",     IF);\n    keywords.put("nil",    NIL);\n    keywords.put("or",     OR);\n    keywords.put("print",  PRINT);\n    keywords.put("return", RETURN);\n    keywords.put("super",  SUPER);\n    keywords.put("this",   THIS);\n    keywords.put("true",   TRUE);\n    keywords.put("var",    VAR);\n    keywords.put("while",  WHILE);\n  }\n
\n
lox/Scanner.java, in class Scanner
\n\n

Then, after we scan an identifier, we check to see if it matches anything in the\nmap.

\n
    while (isAlphaNumeric(peek())) advance();\n\n
lox/Scanner.java
\nin identifier()
\nreplace 1 line
\n
    String text = source.substring(start, current);\n    TokenType type = keywords.get(text);\n    if (type == null) type = IDENTIFIER;\n    addToken(type);\n
  }\n
\n
lox/Scanner.java, in identifier(), replace 1 line
\n\n

If so, we use that keyword’s token type. Otherwise, it’s a regular user-defined\nidentifier.

\n

And with that, we now have a complete scanner for the entire Lox lexical\ngrammar. Fire up the REPL and type in some valid and invalid code. Does it\nproduce the tokens you expect? Try to come up with some interesting edge cases\nand see if it handles them as it should.

\n
\n

Challenges

\n
    \n
  1. \n

    The lexical grammars of Python and Haskell are not regular. What does that\nmean, and why aren’t they?

    \n
    2. \n
  2. \n

    Aside from separating tokens—distinguishing print foo from printfoo—spaces aren’t used for much in most languages. However, in a couple of\ndark corners, a space does affect how code is parsed in CoffeeScript,\nRuby, and the C preprocessor. Where and what effect does it have in each of\nthose languages?

    \n
    3. \n
  3. \n

    Our scanner here, like most, discards comments and whitespace since those\naren’t needed by the parser. Why might you want to write a scanner that does\nnot discard those? What would it be useful for?

    \n
    4. \n
  4. \n

    Add support to Lox’s scanner for C-style /* ... */ block comments. Make\nsure to handle newlines in them. Consider allowing them to nest. Is adding\nsupport for nesting more work than you expected? Why?

    \n
    5. \n
\n
\n
\n

Design Note: Implicit Semicolons

\n

Programmers today are spoiled for choice in languages and have gotten picky\nabout syntax. They want their language to look clean and modern. One bit of\nsyntactic lichen that almost every new language scrapes off (and some ancient\nones like BASIC never had) is ; as an explicit statement terminator.

\n

Instead, they treat a newline as a statement terminator where it makes sense to\ndo so. The “where it makes sense” part is the challenging bit. While most\nstatements are on their own line, sometimes you need to spread a single\nstatement across a couple of lines. Those intermingled newlines should not be\ntreated as terminators.

\n

Most of the obvious cases where the newline should be ignored are easy to\ndetect, but there are a handful of nasty ones:

\n
    \n
  • \n

    A return value on the next line:

    \n
    if (condition) return\n"value"\n
    \n

    Is “value” the value being returned, or do we have a return statement with\n no value followed by an expression statement containing a string literal?

    \n
    • \n
  • \n

    A parenthesized expression on the next line:

    \n
    func\n(parenthesized)\n
    \n

    Is this a call to func(parenthesized), or two expression statements, one\n for func and one for a parenthesized expression?

    \n
    • \n
  • \n

    A - on the next line:

    \n
    first\n-second\n
    \n

    Is this first - second—an infix subtraction—or two expression\n statements, one for first and one to negate second?

    \n
    • \n
\n

In all of these, either treating the newline as a separator or not would both\nproduce valid code, but possibly not the code the user wants. Across languages,\nthere is an unsettling variety of rules used to decide which newlines are\nseparators. Here are a couple:

\n
    \n
  • \n

    Lua completely ignores newlines, but carefully controls its grammar such\nthat no separator between statements is needed at all in most cases. This is\nperfectly legit:

    \n
    a = 1 b = 2\n
    \n

    Lua avoids the return problem by requiring a return statement to be the\nvery last statement in a block. If there is a value after return before\nthe keyword end, it must be for the return. For the other two cases,\nthey allow an explicit ; and expect users to use that. In practice, that\nalmost never happens because there’s no point in a parenthesized or unary\nnegation expression statement.

    \n
    • \n
  • \n

    Go handles newlines in the scanner. If a newline appears following one\nof a handful of token types that are known to potentially end a statement,\nthe newline is treated like a semicolon. Otherwise it is ignored. The Go\nteam provides a canonical code formatter, gofmt, and the ecosystem is\nfervent about its use, which ensures that idiomatic styled code works well\nwith this simple rule.

    \n
    • \n
  • \n

    Python treats all newlines as significant unless an explicit backslash\nis used at the end of a line to continue it to the next line. However,\nnewlines anywhere inside a pair of brackets ((), [], or {}) are\nignored. Idiomatic style strongly prefers the latter.

    \n

    This rule works well for Python because it is a highly statement-oriented\nlanguage. In particular, Python’s grammar ensures a statement never appears\ninside an expression. C does the same, but many other languages which have a\n“lambda” or function literal syntax do not.

    \n

    An example in JavaScript:

    \n
    console.log(function() {\n  statement();\n});\n
    \n

    Here, the console.log() expression contains a function literal which\nin turn contains the statement statement();.

    \n

    Python would need a different set of rules for implicitly joining lines if\nyou could get back into a statement where\nnewlines should become meaningful while still nested inside brackets.

    \n
    • \n
\n\n
    \n
  • \n

    JavaScript’s “automatic semicolon insertion” rule is the real odd\none. Where other languages assume most newlines are meaningful and only a\nfew should be ignored in multi-line statements, JS assumes the opposite. It\ntreats all of your newlines as meaningless whitespace unless it encounters\na parse error. If it does, it goes back and tries turning the previous\nnewline into a semicolon to get something grammatically valid.

    \n

    This design note would turn into a design diatribe if I went into complete\ndetail about how that even works, much less all the various ways that\nJavaScript’s “solution” is a bad idea. It’s a mess. JavaScript is the only\nlanguage I know where many style guides demand explicit semicolons after\nevery statement even though the language theoretically lets you elide them.

    \n
    • \n
\n

If you’re designing a new language, you almost surely should avoid an explicit\nstatement terminator. Programmers are creatures of fashion like other humans, and\nsemicolons are as passé as ALL CAPS KEYWORDS. Just make sure you pick a set of\nrules that make sense for your language’s particular grammar and idioms. And\ndon’t do what JavaScript did.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/script.js", "content": "$(function() {\n $(\"#expand-nav\").click(function() {\n $(\".expandable\").toggleClass(\"shown\");\n });\n\n $(window).scroll(function() {\n var nav = $(\"nav.floating\");\n if ($(window).scrollTop() > 84) {\n nav.addClass(\"pinned\");\n } else {\n nav.removeClass(\"pinned\");\n }\n });\n\n $(window).resize(refreshAsides);\n\n // Since we may not have the height correct for the images, adjust the asides\n // too when an image is loaded.\n $(\"img\").on(\"load\", function() {\n refreshAsides();\n });\n\n // On the off chance the browser supports the new font loader API, use it.\n if (document.fontloader) {\n document.fontloader.notifyWhenFontsReady(function() {\n refreshAsides();\n });\n }\n\n // Lame. Just do another refresh after a second when the font is *probably*\n // loaded to hack around the fact that the metrics changed a bit.\n window.setTimeout(refreshAsides, 200);\n\n refreshAsides();\n});\n\nfunction refreshAsides() {\n $(\"aside\").each(function() {\n var aside = $(this);\n\n // If the asides are inline, clear their position.\n if ($(document).width() <= 48 * 20) {\n aside.css('top', 'auto');\n return;\n }\n\n // Find the span the aside should be anchored next to.\n var name = aside.attr(\"name\");\n if (name == null) {\n window.console.log(\"No name for aside:\");\n window.console.log(aside.context);\n return;\n }\n\n var span = $(\"span[name='\" + name + \"']\");\n if (span == null) {\n window.console.log(\"Could not find span for '\" + name + \"'\");\n return;\n }\n\n // Vertically position the aside next to the span it annotates.\n var pos = span.position();\n if (pos == null) {\n window.console.log(\"Could not find position for '\" + name + \"'\");\n console.log(span);\n return;\n }\n\n if (aside.hasClass(\"bottom\")) {\n aside.offset({top: pos.top + 23 - aside.height()});\n } else {\n aside.offset({top: pos.top - 6});\n }\n });\n}" }, { "path": "site/statements-and-state.html", "content": "\n\n\n\nStatements and State · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
8
\n

Statements and State

\n\n
\n

All my life, my heart has yearned for a thing I cannot name.\nAndré Breton, Mad Love

\n
\n

The interpreter we have so far feels less like programming a real language and\nmore like punching buttons on a calculator. “Programming” to me means building\nup a system out of smaller pieces. We can’t do that yet because we have no way\nto bind a name to some data or function. We can’t compose software without a way\nto refer to the pieces.

\n

To support bindings, our interpreter needs internal state. When you define a\nvariable at the beginning of the program and use it at the end, the interpreter\nhas to hold on to the value of that variable in the meantime. So in this\nchapter, we will give our interpreter a brain that can not just process, but\nremember.

\"A\n

State and statements go hand in hand. Since statements,\nby definition, don’t evaluate to a value, they need to do something else to be\nuseful. That something is called a side effect. It could mean producing\nuser-visible output or modifying some state in the interpreter that can be\ndetected later. The latter makes them a great fit for defining variables or\nother named entities.

\n\n

In this chapter, we’ll do all of that. We’ll define statements that produce\noutput (print) and create state (var). We’ll add expressions to access and\nassign to variables. Finally, we’ll add blocks and local scope. That’s a lot to\nstuff into one chapter, but we’ll chew through it all one bite at a time.

\n

8 . 1Statements

\n

We start by extending Lox’s grammar with statements. They aren’t very different\nfrom expressions. We start with the two simplest kinds:

\n
    \n
  1. \n

    An expression statement lets you place an expression where a statement\nis expected. They exist to evaluate expressions that have side effects. You\nmay not notice them, but you use them all the time in C, Java, and other languages. Any time you see a\nfunction or method call followed by a ;, you’re looking at an expression\nstatement.

    \n
    2. \n
  2. \n

    A print statement evaluates an expression and displays the result to\nthe user. I admit it’s weird to bake printing right into the language\ninstead of making it a library function. Doing so is a concession to the\nfact that we’re building this interpreter one chapter at a time and want to\nbe able to play with it before it’s all done. To make print a library\nfunction, we’d have to wait until we had all of the machinery for defining\nand calling functions before we could witness any\nside effects.

    \n
    3. \n
\n

New syntax means new grammar rules. In this chapter, we finally gain the ability\nto parse an entire Lox script. Since Lox is an imperative, dynamically typed\nlanguage, the “top level” of a script is simply a list of statements. The new\nrules are:

\n
program        → statement* EOF ;\n\nstatement      → exprStmt\n               | printStmt ;\n\nexprStmt       → expression ";" ;\nprintStmt      → "print" expression ";" ;\n
\n

The first rule is now program, which is the starting point for the grammar and\nrepresents a complete Lox script or REPL entry. A program is a list of\nstatements followed by the special “end of file” token. The mandatory end token\nensures the parser consumes the entire input and doesn’t silently ignore\nerroneous unconsumed tokens at the end of a script.

\n

Right now, statement only has two cases for the two kinds of statements we’ve\ndescribed. We’ll fill in more later in this chapter and in the following ones.\nThe next step is turning this grammar into something we can store in memory—syntax trees.

\n

8 . 1 . 1Statement syntax trees

\n

There is no place in the grammar where both an expression and a statement are\nallowed. The operands of, say, + are always expressions, never statements. The\nbody of a while loop is always a statement.

\n

Since the two syntaxes are disjoint, we don’t need a single base class that they\nall inherit from. Splitting expressions and statements into separate class\nhierarchies enables the Java compiler to help us find dumb mistakes like passing\na statement to a Java method that expects an expression.

\n

That means a new base class for statements. As our elders did before us, we will\nuse the cryptic name “Stmt”. With great foresight,\nI have designed our little AST metaprogramming script in anticipation of this.\nThat’s why we passed in “Expr” as a parameter to defineAst(). Now we add\nanother call to define Stmt and its subclasses.

\n\n
      "Unary    : Token operator, Expr right"\n    ));\n
tool/GenerateAst.java
\nin main()
\n
\n\n    defineAst(outputDir, "Stmt", Arrays.asList(\n      "Expression : Expr expression",\n      "Print      : Expr expression"\n    ));\n
  }\n
\n
tool/GenerateAst.java, in main()
\n\n\n

Run the AST generator script and behold the resulting “Stmt.java” file with the\nsyntax tree classes we need for expression and print statements. Don’t forget\nto add the file to your IDE project or makefile or whatever.

\n

8 . 1 . 2Parsing statements

\n

The parser’s parse() method that parses and returns a single expression was a\ntemporary hack to get the last chapter up and running. Now that our grammar has\nthe correct starting rule, program, we can turn parse() into the real deal.

\n
lox/Parser.java
\nmethod parse()
\nreplace 7 lines
\n
  List<Stmt> parse() {\n    List<Stmt> statements = new ArrayList<>();\n    while (!isAtEnd()) {\n      statements.add(statement());\n    }\n\n    return statements; \n  }\n
\n
lox/Parser.java, method parse(), replace 7 lines
\n\n\n

This parses a series of statements, as many as it can find until it hits the end\nof the input. This is a pretty direct translation of the program rule into\nrecursive descent style. We must also chant a minor prayer to the Java verbosity\ngods since we are using ArrayList now.

\n
package com.craftinginterpreters.lox;\n\n
lox/Parser.java
\n
import java.util.ArrayList;\n
import java.util.List;\n
\n
lox/Parser.java
\n\n

A program is a list of statements, and we parse one of those statements using\nthis method:

\n
lox/Parser.java
\nadd after expression()
\n
  private Stmt statement() {\n    if (match(PRINT)) return printStatement();\n\n    return expressionStatement();\n  }\n
\n
lox/Parser.java, add after expression()
\n\n

A little bare bones, but we’ll fill it in with more statement types later. We\ndetermine which specific statement rule is matched by looking at the current\ntoken. A print token means it’s obviously a print statement.

\n

If the next token doesn’t look like any known kind of statement, we assume it\nmust be an expression statement. That’s the typical final fallthrough case when\nparsing a statement, since it’s hard to proactively recognize an expression from\nits first token.

\n

Each statement kind gets its own method. First print:

\n
lox/Parser.java
\nadd after statement()
\n
  private Stmt printStatement() {\n    Expr value = expression();\n    consume(SEMICOLON, "Expect ';' after value.");\n    return new Stmt.Print(value);\n  }\n
\n
lox/Parser.java, add after statement()
\n\n

Since we already matched and consumed the print token itself, we don’t need to\ndo that here. We parse the subsequent expression, consume the terminating\nsemicolon, and emit the syntax tree.

\n

If we didn’t match a print statement, we must have one of these:

\n
lox/Parser.java
\nadd after printStatement()
\n
  private Stmt expressionStatement() {\n    Expr expr = expression();\n    consume(SEMICOLON, "Expect ';' after expression.");\n    return new Stmt.Expression(expr);\n  }\n
\n
lox/Parser.java, add after printStatement()
\n\n

Similar to the previous method, we parse an expression followed by a semicolon.\nWe wrap that Expr in a Stmt of the right type and return it.

\n

8 . 1 . 3Executing statements

\n

We’re running through the previous couple of chapters in microcosm, working our\nway through the front end. Our parser can now produce statement syntax trees, so\nthe next and final step is to interpret them. As in expressions, we use the\nVisitor pattern, but we have a new visitor interface, Stmt.Visitor, to\nimplement since statements have their own base class.

\n

We add that to the list of interfaces Interpreter implements.

\n
lox/Interpreter.java
\nreplace 1 line
\n
class Interpreter implements Expr.Visitor<Object>,\n                             Stmt.Visitor<Void> {\n
  void interpret(Expr expression) { \n
\n
lox/Interpreter.java, replace 1 line
\n\n\n

Unlike expressions, statements produce no values, so the return type of the\nvisit methods is Void, not Object. We have two statement types, and we need a\nvisit method for each. The easiest is expression statements.

\n
lox/Interpreter.java
\nadd after evaluate()
\n
  @Override\n  public Void visitExpressionStmt(Stmt.Expression stmt) {\n    evaluate(stmt.expression);\n    return null;\n  }\n
\n
lox/Interpreter.java, add after evaluate()
\n\n

We evaluate the inner expression using our existing evaluate() method and\ndiscard the value. Then we return null. Java\nrequires that to satisfy the special capitalized Void return type. Weird, but\nwhat can you do?

\n\n

The print statement’s visit method isn’t much different.

\n
lox/Interpreter.java
\nadd after visitExpressionStmt()
\n
  @Override\n  public Void visitPrintStmt(Stmt.Print stmt) {\n    Object value = evaluate(stmt.expression);\n    System.out.println(stringify(value));\n    return null;\n  }\n
\n
lox/Interpreter.java, add after visitExpressionStmt()
\n\n

Before discarding the expression’s value, we convert it to a string using the\nstringify() method we introduced in the last chapter and then dump it to\nstdout.

\n

Our interpreter is able to visit statements now, but we have some work to do to\nfeed them to it. First, modify the old interpret() method in the Interpreter\nclass to accept a list of statements—in other words, a program.

\n
lox/Interpreter.java
\nmethod interpret()
\nreplace 8 lines
\n
  void interpret(List<Stmt> statements) {\n    try {\n      for (Stmt statement : statements) {\n        execute(statement);\n      }\n    } catch (RuntimeError error) {\n      Lox.runtimeError(error);\n    }\n  }\n
\n
lox/Interpreter.java, method interpret(), replace 8 lines
\n\n

This replaces the old code which took a single expression. The new code relies\non this tiny helper method:

\n
lox/Interpreter.java
\nadd after evaluate()
\n
  private void execute(Stmt stmt) {\n    stmt.accept(this);\n  }\n
\n
lox/Interpreter.java, add after evaluate()
\n\n

That’s the statement analogue to the evaluate() method we have for\nexpressions. Since we’re working with lists now, we need to let Java know.

\n
package com.craftinginterpreters.lox;\n
lox/Interpreter.java
\n
\n\nimport java.util.List;\n
\n\nclass Interpreter implements Expr.Visitor<Object>,\n
\n
lox/Interpreter.java
\n\n

The main Lox class is still trying to parse a single expression and pass it to\nthe interpreter. We fix the parsing line like so:

\n
    Parser parser = new Parser(tokens);\n
lox/Lox.java
\nin run()
\nreplace 1 line
\n
    List<Stmt> statements = parser.parse();\n
\n\n    // Stop if there was a syntax error.\n
\n
lox/Lox.java, in run(), replace 1 line
\n\n

And then replace the call to the interpreter with this:

\n
    if (hadError) return;\n\n
lox/Lox.java
\nin run()
\nreplace 1 line
\n
    interpreter.interpret(statements);\n
  }\n
\n
lox/Lox.java, in run(), replace 1 line
\n\n

Basically just plumbing the new syntax through. OK, fire up the interpreter and\ngive it a try. At this point, it’s worth sketching out a little Lox program in a\ntext file to run as a script. Something like:

\n
print "one";\nprint true;\nprint 2 + 1;\n
\n

It almost looks like a real program! Note that the REPL, too, now requires you\nto enter a full statement instead of a simple expression. Don’t forget your\nsemicolons.

\n

8 . 2Global Variables

\n

Now that we have statements, we can start working on state. Before we get into\nall of the complexity of lexical scoping, we’ll start off with the easiest kind\nof variables—globals. We need two new constructs.

\n
    \n
  1. \n

    A variable declaration statement brings a new variable into the world.

    \n
    var beverage = "espresso";\n
    \n

    This creates a new binding that associates a name (here “beverage”) with a\nvalue (here, the string \"espresso\").

    \n
    2. \n
  2. \n

    Once that’s done, a variable expression accesses that binding. When the\nidentifier “beverage” is used as an expression, it looks up the value bound\nto that name and returns it.

    \n
    print beverage; // "espresso".\n
    \n
    3. \n
\n

Later, we’ll add assignment and block scope, but that’s enough to get moving.

\n\n

8 . 2 . 1Variable syntax

\n

As before, we’ll work through the implementation from front to back, starting\nwith the syntax. Variable declarations are statements, but they are different\nfrom other statements, and we’re going to split the statement grammar in two to\nhandle them. That’s because the grammar restricts where some kinds of statements\nare allowed.

\n

The clauses in control flow statements—think the then and else branches of\nan if statement or the body of a while—are each a single statement. But\nthat statement is not allowed to be one that declares a name. This is OK:

\n
if (monday) print "Ugh, already?";\n
\n

But this is not:

\n
if (monday) var beverage = "espresso";\n
\n

We could allow the latter, but it’s confusing. What is the scope of that\nbeverage variable? Does it persist after the if statement? If so, what is\nits value on days other than Monday? Does the variable exist at all on those\ndays?

\n

Code like this is weird, so C, Java, and friends all disallow it. It’s as if\nthere are two levels of “precedence” for statements.\nSome places where a statement is allowed—like inside a block or at the top\nlevel—allow any kind of statement, including declarations. Others allow only\nthe “higher” precedence statements that don’t declare names.

\n\n

To accommodate the distinction, we add another rule for kinds of statements that\ndeclare names.

\n
program        → declaration* EOF ;\n\ndeclaration    → varDecl\n               | statement ;\n\nstatement      → exprStmt\n               | printStmt ;\n
\n

Declaration statements go under the new declaration rule. Right now, it’s only\nvariables, but later it will include functions and classes. Any place where a\ndeclaration is allowed also allows non-declaring statements, so the\ndeclaration rule falls through to statement. Obviously, you can declare\nstuff at the top level of a script, so program routes to the new rule.

\n

The rule for declaring a variable looks like:

\n
varDecl        → "var" IDENTIFIER ( "=" expression )? ";" ;\n
\n

Like most statements, it starts with a leading keyword. In this case, var.\nThen an identifier token for the name of the variable being declared, followed\nby an optional initializer expression. Finally, we put a bow on it with the\nsemicolon.

\n

To access a variable, we define a new kind of primary expression.

\n
primary        → "true" | "false" | "nil"\n               | NUMBER | STRING\n               | "(" expression ")"\n               | IDENTIFIER ;\n
\n

That IDENTIFIER clause matches a single identifier token, which is understood\nto be the name of the variable being accessed.

\n

These new grammar rules get their corresponding syntax trees. Over in the AST\ngenerator, we add a new statement node for a\nvariable declaration.

\n
      "Expression : Expr expression",\n
      "Print      : Expr expression",\n
tool/GenerateAst.java
\nin main()
\nadd “,” to previous line
\n
      "Var        : Token name, Expr initializer"\n
    ));\n
\n
tool/GenerateAst.java, in main(), add “,” to previous line
\n\n\n

It stores the name token so we know what it’s declaring, along with the\ninitializer expression. (If there isn’t an initializer, that field is null.)

\n

Then we add an expression node for accessing a variable.

\n
      "Literal  : Object value",\n
      "Unary    : Token operator, Expr right",\n
tool/GenerateAst.java
\nin main()
\nadd “,” to previous line
\n
      "Variable : Token name"\n
    ));\n
\n
tool/GenerateAst.java, in main(), add “,” to previous line
\n\n

It’s simply a wrapper around the token for the\nvariable name. That’s it. As always, don’t forget to run the AST generator\nscript so that you get updated “Expr.java” and “Stmt.java” files.

\n\n

8 . 2 . 2Parsing variables

\n

Before we parse variable statements, we need to shift around some code to make\nroom for the new declaration rule in the grammar. The top level of a program\nis now a list of declarations, so the entrypoint method to the parser changes.

\n
  List<Stmt> parse() {\n    List<Stmt> statements = new ArrayList<>();\n    while (!isAtEnd()) {\n
lox/Parser.java
\nin parse()
\nreplace 1 line
\n
      statements.add(declaration());\n
    }\n\n    return statements; \n  }\n
\n
lox/Parser.java, in parse(), replace 1 line
\n\n

That calls this new method:

\n
lox/Parser.java
\nadd after expression()
\n
  private Stmt declaration() {\n    try {\n      if (match(VAR)) return varDeclaration();\n\n      return statement();\n    } catch (ParseError error) {\n      synchronize();\n      return null;\n    }\n  }\n
\n
lox/Parser.java, add after expression()
\n\n

Hey, do you remember way back in that earlier chapter when we put the\ninfrastructure in place to do error recovery? We are finally ready to hook that\nup.

\n

This declaration() method is the method we call repeatedly when parsing a\nseries of statements in a block or a script, so it’s the right place to\nsynchronize when the parser goes into panic mode. The whole body of this method\nis wrapped in a try block to catch the exception thrown when the parser begins\nerror recovery. This gets it back to trying to parse the beginning of the next\nstatement or declaration.

\n

The real parsing happens inside the try block. First, it looks to see if we’re\nat a variable declaration by looking for the leading var keyword. If not, it\nfalls through to the existing statement() method that parses print and\nexpression statements.

\n

Remember how statement() tries to parse an expression statement if no other\nstatement matches? And expression() reports a syntax error if it can’t parse\nan expression at the current token? That chain of calls ensures we report an\nerror if a valid declaration or statement isn’t parsed.

\n

When the parser matches a var token, it branches to:

\n
lox/Parser.java
\nadd after printStatement()
\n
  private Stmt varDeclaration() {\n    Token name = consume(IDENTIFIER, "Expect variable name.");\n\n    Expr initializer = null;\n    if (match(EQUAL)) {\n      initializer = expression();\n    }\n\n    consume(SEMICOLON, "Expect ';' after variable declaration.");\n    return new Stmt.Var(name, initializer);\n  }\n
\n
lox/Parser.java, add after printStatement()
\n\n

As always, the recursive descent code follows the grammar rule. The parser has\nalready matched the var token, so next it requires and consumes an identifier\ntoken for the variable name.

\n

Then, if it sees an = token, it knows there is an initializer expression and\nparses it. Otherwise, it leaves the initializer null. Finally, it consumes the\nrequired semicolon at the end of the statement. All this gets wrapped in a\nStmt.Var syntax tree node and we’re groovy.

\n

Parsing a variable expression is even easier. In primary(), we look for an\nidentifier token.

\n
      return new Expr.Literal(previous().literal);\n    }\n
lox/Parser.java
\nin primary()
\n
\n\n    if (match(IDENTIFIER)) {\n      return new Expr.Variable(previous());\n    }\n
\n\n    if (match(LEFT_PAREN)) {\n
\n
lox/Parser.java, in primary()
\n\n

That gives us a working front end for declaring and using variables. All that’s\nleft is to feed it into the interpreter. Before we get to that, we need to talk\nabout where variables live in memory.

\n

8 . 3Environments

\n

The bindings that associate variables to values need to be stored somewhere.\nEver since the Lisp folks invented parentheses, this data structure has been\ncalled an environment.

\"An\n\n

You can think of it like a map where the keys are\nvariable names and the values are the variable’s, uh, values. In fact, that’s\nhow we’ll implement it in Java. We could stuff that map and the code to manage\nit right into Interpreter, but since it forms a nicely delineated concept, we’ll\npull it out into its own class.

\n

Start a new file and add:

\n\n
lox/Environment.java
\ncreate new file
\n
package com.craftinginterpreters.lox;\n\nimport java.util.HashMap;\nimport java.util.Map;\n\nclass Environment {\n  private final Map<String, Object> values = new HashMap<>();\n}\n
\n
lox/Environment.java, create new file
\n\n

There’s a Java Map in there to store the bindings. It uses bare strings for the\nkeys, not tokens. A token represents a unit of code at a specific place in the\nsource text, but when it comes to looking up variables, all identifier tokens\nwith the same name should refer to the same variable (ignoring scope for now).\nUsing the raw string ensures all of those tokens refer to the same map key.

\n

There are two operations we need to support. First, a variable definition binds\na new name to a value.

\n
lox/Environment.java
\nin class Environment
\n
  void define(String name, Object value) {\n    values.put(name, value);\n  }\n
\n
lox/Environment.java, in class Environment
\n\n

Not exactly brain surgery, but we have made one interesting semantic choice.\nWhen we add the key to the map, we don’t check to see if it’s already present.\nThat means that this program works:

\n
var a = "before";\nprint a; // "before".\nvar a = "after";\nprint a; // "after".\n
\n

A variable statement doesn’t just define a new variable, it can also be used\nto redefine an existing variable. We could choose\nto make this an error instead. The user may not intend to redefine an existing\nvariable. (If they did mean to, they probably would have used assignment, not\nvar.) Making redefinition an error would help them find that bug.

\n

However, doing so interacts poorly with the REPL. In the middle of a REPL\nsession, it’s nice to not have to mentally track which variables you’ve already\ndefined. We could allow redefinition in the REPL but not in scripts, but then\nusers would have to learn two sets of rules, and code copied and pasted from one\nform to the other might not work.

\n\n

So, to keep the two modes consistent, we’ll allow it—at least for global\nvariables. Once a variable exists, we need a way to look it up.

\n
class Environment {\n  private final Map<String, Object> values = new HashMap<>();\n
lox/Environment.java
\nin class Environment
\n
\n\n  Object get(Token name) {\n    if (values.containsKey(name.lexeme)) {\n      return values.get(name.lexeme);\n    }\n\n    throw new RuntimeError(name,\n        "Undefined variable '" + name.lexeme + "'.");\n  }\n\n
  void define(String name, Object value) {\n
\n
lox/Environment.java, in class Environment
\n\n

This is a little more semantically interesting. If the variable is found, it\nsimply returns the value bound to it. But what if it’s not? Again, we have a\nchoice:

\n
    \n
  • \n

    Make it a syntax error.

    \n
    • \n
  • \n

    Make it a runtime error.

    \n
    • \n
  • \n

    Allow it and return some default value like nil.

    \n
    • \n
\n

Lox is pretty lax, but the last option is a little too permissive to me.\nMaking it a syntax error—a compile-time error—seems like a smart choice.\nUsing an undefined variable is a bug, and the sooner you detect the mistake, the\nbetter.

\n

The problem is that using a variable isn’t the same as referring to it. You\ncan refer to a variable in a chunk of code without immediately evaluating it if\nthat chunk of code is wrapped inside a function. If we make it a static error to\nmention a variable before it’s been declared, it becomes much harder to define\nrecursive functions.

\n

We could accommodate single recursion—a function that calls itself—by\ndeclaring the function’s own name before we examine its body. But that doesn’t\nhelp with mutually recursive procedures that call each other. Consider:

\n

\n
fun isOdd(n) {\n  if (n == 0) return false;\n  return isEven(n - 1);\n}\n\nfun isEven(n) {\n  if (n == 0) return true;\n  return isOdd(n - 1);\n}\n
\n\n

The isEven() function isn’t defined by the time we\nare looking at the body of isOdd() where it’s called. If we swap the order of\nthe two functions, then isOdd() isn’t defined when we’re looking at\nisEven()’s body.

\n\n

Since making it a static error makes recursive declarations too difficult,\nwe’ll defer the error to runtime. It’s OK to refer to a variable before it’s\ndefined as long as you don’t evaluate the reference. That lets the program\nfor even and odd numbers work, but you’d get a runtime error in:

\n
print a;\nvar a = "too late!";\n
\n

As with type errors in the expression evaluation code, we report a runtime error\nby throwing an exception. The exception contains the variable’s token so we can\ntell the user where in their code they messed up.

\n

8 . 3 . 1Interpreting global variables

\n

The Interpreter class gets an instance of the new Environment class.

\n
class Interpreter implements Expr.Visitor<Object>,\n                             Stmt.Visitor<Void> {\n
lox/Interpreter.java
\nin class Interpreter
\n
  private Environment environment = new Environment();\n\n
  void interpret(List<Stmt> statements) {\n
\n
lox/Interpreter.java, in class Interpreter
\n\n

We store it as a field directly in Interpreter so that the variables stay in\nmemory as long as the interpreter is still running.

\n

We have two new syntax trees, so that’s two new visit methods. The first is for\ndeclaration statements.

\n
lox/Interpreter.java
\nadd after visitPrintStmt()
\n
  @Override\n  public Void visitVarStmt(Stmt.Var stmt) {\n    Object value = null;\n    if (stmt.initializer != null) {\n      value = evaluate(stmt.initializer);\n    }\n\n    environment.define(stmt.name.lexeme, value);\n    return null;\n  }\n
\n
lox/Interpreter.java, add after visitPrintStmt()
\n\n

If the variable has an initializer, we evaluate it. If not, we have another\nchoice to make. We could have made this a syntax error in the parser by\nrequiring an initializer. Most languages don’t, though, so it feels a little\nharsh to do so in Lox.

\n

We could make it a runtime error. We’d let you define an uninitialized variable,\nbut if you accessed it before assigning to it, a runtime error would occur. It’s\nnot a bad idea, but most dynamically typed languages don’t do that. Instead,\nwe’ll keep it simple and say that Lox sets a variable to nil if it isn’t\nexplicitly initialized.

\n
var a;\nprint a; // "nil".\n
\n

Thus, if there isn’t an initializer, we set the value to null, which is the\nJava representation of Lox’s nil value. Then we tell the environment to bind\nthe variable to that value.

\n

Next, we evaluate a variable expression.

\n
lox/Interpreter.java
\nadd after visitUnaryExpr()
\n
  @Override\n  public Object visitVariableExpr(Expr.Variable expr) {\n    return environment.get(expr.name);\n  }\n
\n
lox/Interpreter.java, add after visitUnaryExpr()
\n\n

This simply forwards to the environment which does the heavy lifting to make\nsure the variable is defined. With that, we’ve got rudimentary variables\nworking. Try this out:

\n
var a = 1;\nvar b = 2;\nprint a + b;\n
\n

We can’t reuse code yet, but we can start to build up programs that reuse\ndata.

\n

8 . 4Assignment

\n

It’s possible to create a language that has variables but does not let you\nreassign—or mutate—them. Haskell is one example. SML supports only\nmutable references and arrays—variables cannot be reassigned. Rust steers you\naway from mutation by requiring a mut modifier to enable assignment.

\n

Mutating a variable is a side effect and, as the name suggests, some language\nfolks think side effects are dirty or inelegant. Code\nshould be pure math that produces values—crystalline, unchanging ones—like\nan act of divine creation. Not some grubby automaton that beats blobs of data\ninto shape, one imperative grunt at a time.

\n\n

Lox is not so austere. Lox is an imperative language, and mutation comes with\nthe territory. Adding support for assignment doesn’t require much work. Global\nvariables already support redefinition, so most of the machinery is there now.\nMainly, we’re missing an explicit assignment notation.

\n

8 . 4 . 1Assignment syntax

\n

That little = syntax is more complex than it might seem. Like most C-derived\nlanguages, assignment is an expression and not a\nstatement. As in C, it is the lowest precedence expression form. That means the\nrule slots between expression and equality (the next lowest precedence\nexpression).

\n\n
expression     → assignment ;\nassignment     → IDENTIFIER "=" assignment\n               | equality ;\n
\n

This says an assignment is either an identifier followed by an = and an\nexpression for the value, or an equality (and thus any other) expression.\nLater, assignment will get more complex when we add property setters on\nobjects, like:

\n
instance.field = "value";\n
\n

The easy part is adding the new syntax tree node.

\n
    defineAst(outputDir, "Expr", Arrays.asList(\n
tool/GenerateAst.java
\nin main()
\n
      "Assign   : Token name, Expr value",\n
      "Binary   : Expr left, Token operator, Expr right",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

It has a token for the variable being assigned to, and an expression for the new\nvalue. After you run the AstGenerator to get the new Expr.Assign class, swap out\nthe body of the parser’s existing expression() method to match the updated\nrule.

\n
  private Expr expression() {\n
lox/Parser.java
\nin expression()
\nreplace 1 line
\n
    return assignment();\n
  }\n
\n
lox/Parser.java, in expression(), replace 1 line
\n\n

Here is where it gets tricky. A single token lookahead recursive descent parser\ncan’t see far enough to tell that it’s parsing an assignment until after it\nhas gone through the left-hand side and stumbled onto the =. You might wonder\nwhy it even needs to. After all, we don’t know we’re parsing a + expression\nuntil after we’ve finished parsing the left operand.

\n

The difference is that the left-hand side of an assignment isn’t an expression\nthat evaluates to a value. It’s a sort of pseudo-expression that evaluates to a\n“thing” you can assign to. Consider:

\n
var a = "before";\na = "value";\n
\n

On the second line, we don’t evaluate a (which would return the string\n“before”). We figure out what variable a refers to so we know where to store\nthe right-hand side expression’s value. The classic terms for these\ntwo constructs are l-value and r-value. All\nof the expressions that we’ve seen so far that produce values are r-values. An\nl-value “evaluates” to a storage location that you can assign into.

\n\n

We want the syntax tree to reflect that an l-value isn’t evaluated like a normal\nexpression. That’s why the Expr.Assign node has a Token for the left-hand\nside, not an Expr. The problem is that the parser doesn’t know it’s parsing an\nl-value until it hits the =. In a complex l-value, that may occur many tokens later.

\n
makeList().head.next = node;\n
\n\n

We have only a single token of lookahead, so what do we do? We use a little\ntrick, and it looks like this:

\n
lox/Parser.java
\nadd after expressionStatement()
\n
  private Expr assignment() {\n    Expr expr = equality();\n\n    if (match(EQUAL)) {\n      Token equals = previous();\n      Expr value = assignment();\n\n      if (expr instanceof Expr.Variable) {\n        Token name = ((Expr.Variable)expr).name;\n        return new Expr.Assign(name, value);\n      }\n\n      error(equals, "Invalid assignment target."); \n    }\n\n    return expr;\n  }\n
\n
lox/Parser.java, add after expressionStatement()
\n\n

Most of the code for parsing an assignment expression looks similar to that of\nthe other binary operators like +. We parse the left-hand side, which can be\nany expression of higher precedence. If we find an =, we parse the right-hand\nside and then wrap it all up in an assignment expression tree node.

\n\n

One slight difference from binary operators is that we don’t loop to build up a\nsequence of the same operator. Since assignment is right-associative, we instead\nrecursively call assignment() to parse the right-hand side.

\n

The trick is that right before we create the assignment expression node, we look\nat the left-hand side expression and figure out what kind of assignment target\nit is. We convert the r-value expression node into an l-value representation.

\n

This conversion works because it turns out that every valid assignment target\nhappens to also be valid syntax as a normal\nexpression. Consider a complex field assignment like:

\n\n
newPoint(x + 2, 0).y = 3;\n
\n

The left-hand side of that assignment could also work as a valid expression.

\n
newPoint(x + 2, 0).y;\n
\n

The first example sets the field, the second gets it.

\n

This means we can parse the left-hand side as if it were an expression and\nthen after the fact produce a syntax tree that turns it into an assignment\ntarget. If the left-hand side expression isn’t a valid\nassignment target, we fail with a syntax error. That ensures we report an error\non code like this:

\n
a + b = c;\n
\n\n

Right now, the only valid target is a simple variable expression, but we’ll add\nfields later. The end result of this trick is an assignment expression tree node\nthat knows what it is assigning to and has an expression subtree for the value\nbeing assigned. All with only a single token of lookahead and no backtracking.

\n

8 . 4 . 2Assignment semantics

\n

We have a new syntax tree node, so our interpreter gets a new visit method.

\n
lox/Interpreter.java
\nadd after visitVarStmt()
\n
  @Override\n  public Object visitAssignExpr(Expr.Assign expr) {\n    Object value = evaluate(expr.value);\n    environment.assign(expr.name, value);\n    return value;\n  }\n
\n
lox/Interpreter.java, add after visitVarStmt()
\n\n

For obvious reasons, it’s similar to variable declaration. It evaluates the\nright-hand side to get the value, then stores it in the named variable. Instead\nof using define() on Environment, it calls this new method:

\n
lox/Environment.java
\nadd after get()
\n
  void assign(Token name, Object value) {\n    if (values.containsKey(name.lexeme)) {\n      values.put(name.lexeme, value);\n      return;\n    }\n\n    throw new RuntimeError(name,\n        "Undefined variable '" + name.lexeme + "'.");\n  }\n
\n
lox/Environment.java, add after get()
\n\n

The key difference between assignment and definition is that assignment is not\nallowed to create a new variable. In terms of our\nimplementation, that means it’s a runtime error if the key doesn’t already exist\nin the environment’s variable map.

\n\n

The last thing the visit() method does is return the assigned value. That’s\nbecause assignment is an expression that can be nested inside other expressions,\nlike so:

\n
var a = 1;\nprint a = 2; // "2".\n
\n

Our interpreter can now create, read, and modify variables. It’s about as\nsophisticated as early BASICs. Global variables are\nsimple, but writing a large program when any two chunks of code can accidentally\nstep on each other’s state is no fun. We want local variables, which means\nit’s time for scope.

\n\n

8 . 5Scope

\n

A scope defines a region where a name maps to a certain entity. Multiple\nscopes enable the same name to refer to different things in different contexts.\nIn my house, “Bob” usually refers to me. But maybe in your town you know a\ndifferent Bob. Same name, but different dudes based on where you say it.

\n

Lexical scope (or the less commonly heard\nstatic scope) is a specific style of scoping where the text of the program\nitself shows where a scope begins and ends. In Lox, as in most modern languages,\nvariables are lexically scoped. When you see an expression that uses some\nvariable, you can figure out which variable declaration it refers to just by\nstatically reading the code.

\n\n

For example:

\n
{\n  var a = "first";\n  print a; // "first".\n}\n\n{\n  var a = "second";\n  print a; // "second".\n}\n
\n

Here, we have two blocks with a variable a declared in each of them. You and\nI can tell just from looking at the code that the use of a in the first\nprint statement refers to the first a, and the second one refers to the\nsecond.

\"An\n

This is in contrast to dynamic scope where you don’t know what a name refers\nto until you execute the code. Lox doesn’t have dynamically scoped variables,\nbut methods and fields on objects are dynamically scoped.

\n
class Saxophone {\n  play() {\n    print "Careless Whisper";\n  }\n}\n\nclass GolfClub {\n  play() {\n    print "Fore!";\n  }\n}\n\nfun playIt(thing) {\n  thing.play();\n}\n
\n

When playIt() calls thing.play(), we don’t know if we’re about to hear\n“Careless Whisper” or “Fore!” It depends on whether you pass a Saxophone or a\nGolfClub to the function, and we don’t know that until runtime.

\n

Scope and environments are close cousins. The former is the theoretical concept,\nand the latter is the machinery that implements it. As our interpreter works its\nway through code, syntax tree nodes that affect scope will change the\nenvironment. In a C-ish syntax like Lox’s, scope is controlled by curly-braced\nblocks. (That’s why we call it block scope.)

\n
{\n  var a = "in block";\n}\nprint a; // Error! No more "a".\n
\n

The beginning of a block introduces a new local scope, and that scope ends when\nexecution passes the closing }. Any variables declared inside the block\ndisappear.

\n

8 . 5 . 1Nesting and shadowing

\n

A first cut at implementing block scope might work like this:

\n
    \n
  1. \n

    As we visit each statement inside the block, keep track of any variables\ndeclared.

    \n
    2. \n
  2. \n

    After the last statement is executed, tell the environment to delete all of\nthose variables.

    \n
    3. \n
\n

That would work for the previous example. But remember, one motivation for\nlocal scope is encapsulation—a block of code in one corner of the program\nshouldn’t interfere with some other block. Check this out:

\n
// How loud?\nvar volume = 11;\n\n// Silence.\nvolume = 0;\n\n// Calculate size of 3x4x5 cuboid.\n{\n  var volume = 3 * 4 * 5;\n  print volume;\n}\n
\n

Look at the block where we calculate the volume of the cuboid using a local\ndeclaration of volume. After the block exits, the interpreter will delete the\nglobal volume variable. That ain’t right. When we exit the block, we should\nremove any variables declared inside the block, but if there is a variable with\nthe same name declared outside of the block, that’s a different variable. It\nshouldn’t get touched.

\n

When a local variable has the same name as a variable in an enclosing scope, it\nshadows the outer one. Code inside the block can’t see it any more—it is\nhidden in the “shadow” cast by the inner one—but it’s still there.

\n

When we enter a new block scope, we need to preserve variables defined in outer\nscopes so they are still around when we exit the inner block. We do that by\ndefining a fresh environment for each block containing only the variables\ndefined in that scope. When we exit the block, we discard its environment and\nrestore the previous one.

\n

We also need to handle enclosing variables that are not shadowed.

\n
var global = "outside";\n{\n  var local = "inside";\n  print global + local;\n}\n
\n

Here, global lives in the outer global environment and local is defined\ninside the block’s environment. In that print statement, both of those\nvariables are in scope. In order to find them, the interpreter must search not\nonly the current innermost environment, but also any enclosing ones.

\n

We implement this by chaining the environments\ntogether. Each environment has a reference to the environment of the immediately\nenclosing scope. When we look up a variable, we walk that chain from innermost\nout until we find the variable. Starting at the inner scope is how we make local\nvariables shadow outer ones.

\"Environments\n\n

Before we add block syntax to the grammar, we’ll beef up our Environment class\nwith support for this nesting. First, we give each environment a reference to\nits enclosing one.

\n
class Environment {\n
lox/Environment.java
\nin class Environment
\n
  final Environment enclosing;\n
  private final Map<String, Object> values = new HashMap<>();\n
\n
lox/Environment.java, in class Environment
\n\n

This field needs to be initialized, so we add a couple of constructors.

\n
lox/Environment.java
\nin class Environment
\n
  Environment() {\n    enclosing = null;\n  }\n\n  Environment(Environment enclosing) {\n    this.enclosing = enclosing;\n  }\n
\n
lox/Environment.java, in class Environment
\n\n

The no-argument constructor is for the global scope’s environment, which ends\nthe chain. The other constructor creates a new local scope nested inside the\ngiven outer one.

\n

We don’t have to touch the define() method—a new variable is always\ndeclared in the current innermost scope. But variable lookup and assignment work\nwith existing variables and they need to walk the chain to find them. First,\nlookup:

\n
      return values.get(name.lexeme);\n    }\n
lox/Environment.java
\nin get()
\n
\n\n    if (enclosing != null) return enclosing.get(name);\n
\n\n    throw new RuntimeError(name,\n        "Undefined variable '" + name.lexeme + "'.");\n
\n
lox/Environment.java, in get()
\n\n

If the variable isn’t found in this environment, we simply try the enclosing\none. That in turn does the same thing recursively,\nso this will ultimately walk the entire chain. If we reach an environment with\nno enclosing one and still don’t find the variable, then we give up and report\nan error as before.

\n

Assignment works the same way.

\n\n
      values.put(name.lexeme, value);\n      return;\n    }\n\n
lox/Environment.java
\nin assign()
\n
    if (enclosing != null) {\n      enclosing.assign(name, value);\n      return;\n    }\n\n
    throw new RuntimeError(name,\n
\n
lox/Environment.java, in assign()
\n\n

Again, if the variable isn’t in this environment, it checks the outer one,\nrecursively.

\n

8 . 5 . 2Block syntax and semantics

\n

Now that Environments nest, we’re ready to add blocks to the language. Behold\nthe grammar:

\n
statement      → exprStmt\n               | printStmt\n               | block ;\n\nblock          → "{" declaration* "}" ;\n
\n

A block is a (possibly empty) series of statements or declarations surrounded by\ncurly braces. A block is itself a statement and can appear anywhere a statement\nis allowed. The syntax tree node looks like this:

\n
    defineAst(outputDir, "Stmt", Arrays.asList(\n
tool/GenerateAst.java
\nin main()
\n
      "Block      : List<Stmt> statements",\n
      "Expression : Expr expression",\n
\n
tool/GenerateAst.java, in main()
\n\n\n

It contains the list of statements that are inside\nthe block. Parsing is straightforward. Like other statements, we detect the\nbeginning of a block by its leading token—in this case the {. In the\nstatement() method, we add:

\n\n
    if (match(PRINT)) return printStatement();\n
lox/Parser.java
\nin statement()
\n
    if (match(LEFT_BRACE)) return new Stmt.Block(block());\n
\n\n    return expressionStatement();\n
\n
lox/Parser.java, in statement()
\n\n

All the real work happens here:

\n
lox/Parser.java
\nadd after expressionStatement()
\n
  private List<Stmt> block() {\n    List<Stmt> statements = new ArrayList<>();\n\n    while (!check(RIGHT_BRACE) && !isAtEnd()) {\n      statements.add(declaration());\n    }\n\n    consume(RIGHT_BRACE, "Expect '}' after block.");\n    return statements;\n  }\n
\n
lox/Parser.java, add after expressionStatement()
\n\n

We create an empty list and then parse statements and\nadd them to the list until we reach the end of the block, marked by the closing\n}. Note that the loop also has an explicit check for isAtEnd(). We have to\nbe careful to avoid infinite loops, even when parsing invalid code. If the user\nforgets a closing }, the parser needs to not get stuck.

\n\n

That’s it for syntax. For semantics, we add another visit method to Interpreter.

\n
lox/Interpreter.java
\nadd after execute()
\n
  @Override\n  public Void visitBlockStmt(Stmt.Block stmt) {\n    executeBlock(stmt.statements, new Environment(environment));\n    return null;\n  }\n
\n
lox/Interpreter.java, add after execute()
\n\n

To execute a block, we create a new environment for the block’s scope and pass\nit off to this other method:

\n
lox/Interpreter.java
\nadd after execute()
\n
  void executeBlock(List<Stmt> statements,\n                    Environment environment) {\n    Environment previous = this.environment;\n    try {\n      this.environment = environment;\n\n      for (Stmt statement : statements) {\n        execute(statement);\n      }\n    } finally {\n      this.environment = previous;\n    }\n  }\n
\n
lox/Interpreter.java, add after execute()
\n\n

This new method executes a list of statements in the context of a given environment. Up until now, the environment field in\nInterpreter always pointed to the same environment—the global one. Now, that\nfield represents the current environment. That’s the environment that\ncorresponds to the innermost scope containing the code to be executed.

\n

To execute code within a given scope, this method updates the interpreter’s\nenvironment field, visits all of the statements, and then restores the\nprevious value. As is always good practice in Java, it restores the previous\nenvironment using a finally clause. That way it gets restored even if an\nexception is thrown.

\n\n

Surprisingly, that’s all we need to do in order to fully support local\nvariables, nesting, and shadowing. Go ahead and try this out:

\n
var a = "global a";\nvar b = "global b";\nvar c = "global c";\n{\n  var a = "outer a";\n  var b = "outer b";\n  {\n    var a = "inner a";\n    print a;\n    print b;\n    print c;\n  }\n  print a;\n  print b;\n  print c;\n}\nprint a;\nprint b;\nprint c;\n
\n

Our little interpreter can remember things now. We are inching closer to\nsomething resembling a full-featured programming language.

\n
\n

Challenges

\n
    \n
  1. \n

    The REPL no longer supports entering a single expression and automatically\nprinting its result value. That’s a drag. Add support to the REPL to let\nusers type in both statements and expressions. If they enter a statement,\nexecute it. If they enter an expression, evaluate it and display the result\nvalue.

    \n
    2. \n
  2. \n

    Maybe you want Lox to be a little more explicit about variable\ninitialization. Instead of implicitly initializing variables to nil, make\nit a runtime error to access a variable that has not been initialized or\nassigned to, as in:

    \n
    // No initializers.\nvar a;\nvar b;\n\na = "assigned";\nprint a; // OK, was assigned first.\n\nprint b; // Error!\n
    \n
    3. \n
  3. \n

    What does the following program do?

    \n
    var a = 1;\n{\n  var a = a + 2;\n  print a;\n}\n
    \n

    What did you expect it to do? Is it what you think it should do? What\ndoes analogous code in other languages you are familiar with do? What do\nyou think users will expect this to do?

    \n
    4. \n
\n
\n
\n

Design Note: Implicit Variable Declaration

\n

Lox has distinct syntax for declaring a new variable and assigning to an\nexisting one. Some languages collapse those to only assignment syntax. Assigning\nto a non-existent variable automatically brings it into being. This is called\nimplicit variable declaration and exists in Python, Ruby, and CoffeeScript,\namong others. JavaScript has an explicit syntax to declare variables, but can\nalso create new variables on assignment. Visual Basic has an option to enable\nor disable implicit variables.

\n

When the same syntax can assign or create a variable, each language must decide\nwhat happens when it isn’t clear about which behavior the user intends. In\nparticular, each language must choose how implicit declaration interacts with\nshadowing, and which scope an implicitly declared variable goes into.

\n
    \n
  • \n

    In Python, assignment always creates a variable in the current function’s\nscope, even if there is a variable with the same name declared outside of\nthe function.

    \n
    • \n
  • \n

    Ruby avoids some ambiguity by having different naming rules for local and\nglobal variables. However, blocks in Ruby (which are more like closures than\nlike “blocks” in C) have their own scope, so it still has the problem.\nAssignment in Ruby assigns to an existing variable outside of the current\nblock if there is one with the same name. Otherwise, it creates a new\nvariable in the current block’s scope.

    \n
    • \n
  • \n

    CoffeeScript, which takes after Ruby in many ways, is similar. It explicitly\ndisallows shadowing by saying that assignment always assigns to a variable\nin an outer scope if there is one, all the way up to the outermost global\nscope. Otherwise, it creates the variable in the current function scope.

    \n
    • \n
  • \n

    In JavaScript, assignment modifies an existing variable in any enclosing\nscope, if found. If not, it implicitly creates a new variable in the\nglobal scope.

    \n
    • \n
\n

The main advantage to implicit declaration is simplicity. There’s less syntax\nand no “declaration” concept to learn. Users can just start assigning stuff and\nthe language figures it out.

\n

Older, statically typed languages like C benefit from explicit declaration\nbecause they give the user a place to tell the compiler what type each variable\nhas and how much storage to allocate for it. In a dynamically typed,\ngarbage-collected language, that isn’t really necessary, so you can get away\nwith making declarations implicit. It feels a little more “scripty”, more “you\nknow what I mean”.

\n

But is that a good idea? Implicit declaration has some problems.

\n
    \n
  • \n

    A user may intend to assign to an existing variable, but may have misspelled\nit. The interpreter doesn’t know that, so it goes ahead and silently creates\nsome new variable and the variable the user wanted to assign to still has\nits old value. This is particularly heinous in JavaScript where a typo will\ncreate a global variable, which may in turn interfere with other code.

    \n
    • \n
  • \n

    JS, Ruby, and CoffeeScript use the presence of an existing variable with the\nsame name—even in an outer scope—to determine whether or not an\nassignment creates a new variable or assigns to an existing one. That means\nadding a new variable in a surrounding scope can change the meaning of\nexisting code. What was once a local variable may silently turn into an\nassignment to that new outer variable.

    \n
    • \n
  • \n

    In Python, you may want to assign to some variable outside of the current\nfunction instead of creating a new variable in the current one, but you\ncan’t.

    \n
    • \n
\n

Over time, the languages I know with implicit variable declaration ended up\nadding more features and complexity to deal with these problems.

\n
    \n
  • \n

    Implicit declaration of global variables in JavaScript is universally\nconsidered a mistake today. “Strict mode” disables it and makes it a compile\nerror.

    \n
    • \n
  • \n

    Python added a global statement to let you explicitly assign to a global\nvariable from within a function. Later, as functional programming and nested\nfunctions became more popular, they added a similar nonlocal statement to\nassign to variables in enclosing functions.

    \n
    • \n
  • \n

    Ruby extended its block syntax to allow declaring certain variables to be\nexplicitly local to the block even if the same name exists in an outer\nscope.

    \n
    • \n
\n

Given those, I think the simplicity argument is mostly lost. There is an\nargument that implicit declaration is the right default but I personally find\nthat less compelling.

\n

My opinion is that implicit declaration made sense in years past when most\nscripting languages were heavily imperative and code was pretty flat. As\nprogrammers have gotten more comfortable with deep nesting, functional\nprogramming, and closures, it’s become much more common to want access to\nvariables in outer scopes. That makes it more likely that users will run into\nthe tricky cases where it’s not clear whether they intend their assignment to\ncreate a new variable or reuse a surrounding one.

\n

So I prefer explicitly declaring variables, which is why Lox requires it.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/strings.html", "content": "\n\n\n\nStrings · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
19
\n

Strings

\n\n
\n

“Ah? A small aversion to menial labor?” The doctor cocked an eyebrow.\n“Understandable, but misplaced. One should treasure those hum-drum\ntasks that keep the body occupied but leave the mind and heart unfettered.”

\n

Tad Williams, The Dragonbone Chair

\n
\n

Our little VM can represent three types of values right now: numbers, Booleans,\nand nil. Those types have two important things in common: they’re immutable\nand they’re small. Numbers are the largest, and they still fit into two 64-bit\nwords. That’s a small enough price that we can afford to pay it for all values,\neven Booleans and nils which don’t need that much space.

\n

Strings, unfortunately, are not so petite. There’s no maximum length for a\nstring. Even if we were to artificially cap it at some contrived limit like\n255 characters, that’s still too much memory to spend\non every single value.

\n\n

We need a way to support values whose sizes vary, sometimes greatly. This is\nexactly what dynamic allocation on the heap is designed for. We can allocate as\nmany bytes as we need. We get back a pointer that we’ll use to keep track of the\nvalue as it flows through the VM.

\n

19 . 1Values and Objects

\n

Using the heap for larger, variable-sized values and the stack for smaller,\natomic ones leads to a two-level representation. Every Lox value that you can\nstore in a variable or return from an expression will be a Value. For small,\nfixed-size types like numbers, the payload is stored directly inside the Value\nstruct itself.

\n

If the object is larger, its data lives on the heap. Then the Value’s payload is\na pointer to that blob of memory. We’ll eventually have a handful of\nheap-allocated types in clox: strings, instances, functions, you get the idea.\nEach type has its own unique data, but there is also state they all share that\nour future garbage collector will use to manage their memory.

\"Field\n

We’ll call this common representation “Obj”. Each Lox\nvalue whose state lives on the heap is an Obj. We can thus use a single new\nValueType case to refer to all heap-allocated types.

\n\n
  VAL_NUMBER,\n
value.h
\nin enum ValueType
\n
  VAL_OBJ\n
} ValueType;\n
\n
value.h, in enum ValueType
\n\n

When a Value’s type is VAL_OBJ, the payload is a pointer to the heap memory,\nso we add another case to the union for that.

\n
    double number;\n
value.h
\nin struct Value
\n
    Obj* obj;\n
  } as; \n
\n
value.h, in struct Value
\n\n

As we did with the other value types, we crank out a couple of helpful macros\nfor working with Obj values.

\n
#define IS_NUMBER(value)  ((value).type == VAL_NUMBER)\n
value.h
\nadd after struct Value
\n
#define IS_OBJ(value)     ((value).type == VAL_OBJ)\n
\n\n#define AS_BOOL(value)    ((value).as.boolean)\n
\n
value.h, add after struct Value
\n\n

This evaluates to true if the given Value is an Obj. If so, we can use this:

\n
#define IS_OBJ(value)     ((value).type == VAL_OBJ)\n\n
value.h
\n
#define AS_OBJ(value)     ((value).as.obj)\n
#define AS_BOOL(value)    ((value).as.boolean)\n
\n
value.h
\n\n

It extracts the Obj pointer from the value. We can also go the other way.

\n
#define NUMBER_VAL(value) ((Value){VAL_NUMBER, {.number = value}})\n
value.h
\n
#define OBJ_VAL(object)   ((Value){VAL_OBJ, {.obj = (Obj*)object}})\n
\n\ntypedef struct {\n
\n
value.h
\n\n

This takes a bare Obj pointer and wraps it in a full Value.

\n

19 . 2Struct Inheritance

\n

Every heap-allocated value is an Obj, but Objs are\nnot all the same. For strings, we need the array of characters. When we get to\ninstances, they will need their data fields. A function object will need its\nchunk of bytecode. How do we handle different payloads and sizes? We can’t use\nanother union like we did for Value since the sizes are all over the place.

\n\n

Instead, we’ll use another technique. It’s been around for ages, to the point\nthat the C specification carves out specific support for it, but I don’t know\nthat it has a canonical name. It’s an example of type punning, but that\nterm is too broad. In the absence of any better ideas, I’ll call it struct\ninheritance, because it relies on structs and roughly follows how\nsingle-inheritance of state works in object-oriented languages.

\n

Like a tagged union, each Obj starts with a tag field that identifies what kind\nof object it is—string, instance, etc. Following that are the payload fields.\nInstead of a union with cases for each type, each type is its own separate\nstruct. The tricky part is how to treat these structs uniformly since C has no\nconcept of inheritance or polymorphism. I’ll explain that soon, but first lets\nget the preliminary stuff out of the way.

\n

The name “Obj” itself refers to a struct that contains the state shared across\nall object types. It’s sort of like the “base class” for objects. Because of\nsome cyclic dependencies between values and objects, we forward-declare it in\nthe “value” module.

\n
#include "common.h"\n\n
value.h
\n
typedef struct Obj Obj;\n\n
typedef enum {\n
\n
value.h
\n\n

And the actual definition is in a new module.

\n
object.h
\ncreate new file
\n
#ifndef clox_object_h\n#define clox_object_h\n\n#include "common.h"\n#include "value.h"\n\nstruct Obj {\n  ObjType type;\n};\n\n#endif\n
\n
object.h, create new file
\n\n

Right now, it contains only the type tag. Shortly, we’ll add some other\nbookkeeping information for memory management. The type enum is this:

\n
#include "value.h"\n
object.h
\n
\n\ntypedef enum {\n  OBJ_STRING,\n} ObjType;\n
\n\nstruct Obj {\n
\n
object.h
\n\n

Obviously, that will be more useful in later chapters after we add more\nheap-allocated types. Since we’ll be accessing these tag types frequently, it’s\nworth making a little macro that extracts the object type tag from a given\nValue.

\n
#include "value.h"\n
object.h
\n
\n\n#define OBJ_TYPE(value)        (AS_OBJ(value)->type)\n
\n\ntypedef enum {\n
\n
object.h
\n\n

That’s our foundation.

\n

Now, let’s build strings on top of it. The payload for strings is defined in a\nseparate struct. Again, we need to forward-declare it.

\n
typedef struct Obj Obj;\n
value.h
\n
typedef struct ObjString ObjString;\n
\n\ntypedef enum {\n
\n
value.h
\n\n

The definition lives alongside Obj.

\n
};\n
object.h
\nadd after struct Obj
\n
\n\nstruct ObjString {\n  Obj obj;\n  int length;\n  char* chars;\n};\n
\n\n#endif\n
\n
object.h, add after struct Obj
\n\n

A string object contains an array of characters. Those are stored in a separate,\nheap-allocated array so that we set aside only as much room as needed for each\nstring. We also store the number of bytes in the array. This isn’t strictly\nnecessary but lets us tell how much memory is allocated for the string without\nwalking the character array to find the null terminator.

\n

Because ObjString is an Obj, it also needs the state all Objs share. It\naccomplishes that by having its first field be an Obj. C specifies that struct\nfields are arranged in memory in the order that they are declared. Also, when\nyou nest structs, the inner struct’s fields are expanded right in place. So the\nmemory for Obj and for ObjString looks like this:

\"The\n

Note how the first bytes of ObjString exactly line up with Obj. This is not a\ncoincidence—C mandates it. This is designed to\nenable a clever pattern: You can take a pointer to a struct and safely convert\nit to a pointer to its first field and back.

\n\n

Given an ObjString*, you can safely cast it to Obj* and then access the\ntype field from it. Every ObjString “is” an Obj in the OOP sense of “is”. When\nwe later add other object types, each struct will have an Obj as its first\nfield. Any code that wants to work with all objects can treat them as base\nObj* and ignore any other fields that may happen to follow.

\n

You can go in the other direction too. Given an Obj*, you can “downcast” it to\nan ObjString*. Of course, you need to ensure that the Obj* pointer you have\ndoes point to the obj field of an actual ObjString. Otherwise, you are\nunsafely reinterpreting random bits of memory. To detect that such a cast is\nsafe, we add another macro.

\n
#define OBJ_TYPE(value)        (AS_OBJ(value)->type)\n
object.h
\n
\n\n#define IS_STRING(value)       isObjType(value, OBJ_STRING)\n
\n\ntypedef enum {\n
\n
object.h
\n\n

It takes a Value, not a raw Obj* because most code in the VM works with\nValues. It relies on this inline function:

\n
};\n\n
object.h
\nadd after struct ObjString
\n
static inline bool isObjType(Value value, ObjType type) {\n  return IS_OBJ(value) && AS_OBJ(value)->type == type;\n}\n\n
#endif\n
\n
object.h, add after struct ObjString
\n\n

Pop quiz: Why not just put the body of this function right in the macro? What’s\ndifferent about this one compared to the others? Right, it’s because the body\nuses value twice. A macro is expanded by inserting the argument expression\nevery place the parameter name appears in the body. If a macro uses a parameter\nmore than once, that expression gets evaluated multiple times.

\n

That’s bad if the expression has side effects. If we put the body of\nisObjType() into the macro definition and then you did, say,

\n
IS_STRING(POP())\n
\n

then it would pop two values off the stack! Using a function fixes that.

\n

As long as we ensure that we set the type tag correctly whenever we create an\nObj of some type, this macro will tell us when it’s safe to cast a value to a\nspecific object type. We can do that using these:

\n
#define IS_STRING(value)       isObjType(value, OBJ_STRING)\n
object.h
\n
\n\n#define AS_STRING(value)       ((ObjString*)AS_OBJ(value))\n#define AS_CSTRING(value)      (((ObjString*)AS_OBJ(value))->chars)\n
\n\ntypedef enum {\n
\n
object.h
\n\n

These two macros take a Value that is expected to contain a pointer to a valid\nObjString on the heap. The first one returns the ObjString* pointer. The\nsecond one steps through that to return the character array itself, since that’s\noften what we’ll end up needing.

\n

19 . 3Strings

\n

OK, our VM can now represent string values. It’s time to add strings to the\nlanguage itself. As usual, we begin in the front end. The lexer already\ntokenizes string literals, so it’s the parser’s turn.

\n
  [TOKEN_IDENTIFIER]    = {NULL,     NULL,   PREC_NONE},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_STRING]        = {string,   NULL,   PREC_NONE},\n
  [TOKEN_NUMBER]        = {number,   NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n

When the parser hits a string token, it calls this parse function:

\n
compiler.c
\nadd after number()
\n
static void string() {\n  emitConstant(OBJ_VAL(copyString(parser.previous.start + 1,\n                                  parser.previous.length - 2)));\n}\n
\n
compiler.c, add after number()
\n\n

This takes the string’s characters directly from the\nlexeme. The + 1 and - 2 parts trim the leading and trailing quotation marks.\nIt then creates a string object, wraps it in a Value, and stuffs it into the\nconstant table.

\n\n

To create the string, we use copyString(), which is declared in object.h.

\n
};\n\n
object.h
\nadd after struct ObjString
\n
ObjString* copyString(const char* chars, int length);\n\n
static inline bool isObjType(Value value, ObjType type) {\n
\n
object.h, add after struct ObjString
\n\n

The compiler module needs to include that.

\n
#define clox_compiler_h\n\n
compiler.h
\n
#include "object.h"\n
#include "vm.h"\n
\n
compiler.h
\n\n

Our “object” module gets an implementation file where we define the new\nfunction.

\n
object.c
\ncreate new file
\n
#include <stdio.h>\n#include <string.h>\n\n#include "memory.h"\n#include "object.h"\n#include "value.h"\n#include "vm.h"\n\nObjString* copyString(const char* chars, int length) {\n  char* heapChars = ALLOCATE(char, length + 1);\n  memcpy(heapChars, chars, length);\n  heapChars[length] = '\\0';\n  return allocateString(heapChars, length);\n}\n
\n
object.c, create new file
\n\n

First, we allocate a new array on the heap, just big enough for the string’s\ncharacters and the trailing terminator, using\nthis low-level macro that allocates an array with a given element type and\ncount:

\n
#include "common.h"\n\n
memory.h
\n
#define ALLOCATE(type, count) \\\n    (type*)reallocate(NULL, 0, sizeof(type) * (count))\n\n
#define GROW_CAPACITY(capacity) \\\n
\n
memory.h
\n\n

Once we have the array, we copy over the characters from the lexeme and\nterminate it.

\n\n

You might wonder why the ObjString can’t just point back to the original\ncharacters in the source string. Some ObjStrings will be created dynamically at\nruntime as a result of string operations like concatenation. Those strings\nobviously need to dynamically allocate memory for the characters, which means\nthe string needs to free that memory when it’s no longer needed.

\n

If we had an ObjString for a string literal, and tried to free its character\narray that pointed into the original source code string, bad things would\nhappen. So, for literals, we preemptively copy the characters over to the heap.\nThis way, every ObjString reliably owns its character array and can free it.

\n

The real work of creating a string object happens in this function:

\n
#include "vm.h"\n\n
object.c
\n
static ObjString* allocateString(char* chars, int length) {\n  ObjString* string = ALLOCATE_OBJ(ObjString, OBJ_STRING);\n  string->length = length;\n  string->chars = chars;\n  return string;\n}\n
\n
object.c
\n\n

It creates a new ObjString on the heap and then initializes its fields. It’s\nsort of like a constructor in an OOP language. As such, it first calls the “base\nclass” constructor to initialize the Obj state, using a new macro.

\n
#include "vm.h"\n
object.c
\n
\n\n#define ALLOCATE_OBJ(type, objectType) \\\n    (type*)allocateObject(sizeof(type), objectType)\n
\n\nstatic ObjString* allocateString(char* chars, int length) {\n
\n
object.c
\n\n

Like the previous macro, this exists mainly to\navoid the need to redundantly cast a void* back to the desired type. The\nactual functionality is here:

\n\n
#define ALLOCATE_OBJ(type, objectType) \\\n    (type*)allocateObject(sizeof(type), objectType)\n
object.c
\n
\n\nstatic Obj* allocateObject(size_t size, ObjType type) {\n  Obj* object = (Obj*)reallocate(NULL, 0, size);\n  object->type = type;\n  return object;\n}\n
\n\nstatic ObjString* allocateString(char* chars, int length) {\n
\n
object.c
\n\n

It allocates an object of the given size on the heap. Note that the size is\nnot just the size of Obj itself. The caller passes in the number of bytes so\nthat there is room for the extra payload fields needed by the specific object\ntype being created.

\n

Then it initializes the Obj state—right now, that’s just the type tag. This\nfunction returns to allocateString(), which finishes initializing the ObjString\nfields. Voilà, we can compile and execute string\nliterals.

\n\n

19 . 4Operations on Strings

\n

Our fancy strings are there, but they don’t do much of anything yet. A good\nfirst step is to make the existing print code not barf on the new value type.

\n
    case VAL_NUMBER: printf("%g", AS_NUMBER(value)); break;\n
value.c
\nin printValue()
\n
    case VAL_OBJ: printObject(value); break;\n
  }\n
\n
value.c, in printValue()
\n\n

If the value is a heap-allocated object, it defers to a helper function over in\nthe “object” module.

\n
ObjString* copyString(const char* chars, int length);\n
object.h
\nadd after copyString()
\n
void printObject(Value value);\n
\n\nstatic inline bool isObjType(Value value, ObjType type) {\n
\n
object.h, add after copyString()
\n\n

The implementation looks like this:

\n
object.c
\nadd after copyString()
\n
void printObject(Value value) {\n  switch (OBJ_TYPE(value)) {\n    case OBJ_STRING:\n      printf("%s", AS_CSTRING(value));\n      break;\n  }\n}\n
\n
object.c, add after copyString()
\n\n

We have only a single object type now, but this function will sprout additional\nswitch cases in later chapters. For string objects, it simply prints the character array as a C string.

\n\n

The equality operators also need to gracefully handle strings. Consider:

\n
"string" == "string"\n
\n

These are two separate string literals. The compiler will make two separate\ncalls to copyString(), create two distinct ObjString objects and store them as\ntwo constants in the chunk. They are different objects in the heap. But our\nusers (and thus we) expect strings to have value equality. The above expression\nshould evaluate to true. That requires a little special support.

\n
    case VAL_NUMBER: return AS_NUMBER(a) == AS_NUMBER(b);\n
value.c
\nin valuesEqual()
\n
    case VAL_OBJ: {\n      ObjString* aString = AS_STRING(a);\n      ObjString* bString = AS_STRING(b);\n      return aString->length == bString->length &&\n          memcmp(aString->chars, bString->chars,\n                 aString->length) == 0;\n    }\n
    default:         return false; // Unreachable.\n
\n
value.c, in valuesEqual()
\n\n

If the two values are both strings, then they are equal if their character\narrays contain the same characters, regardless of whether they are two separate\nobjects or the exact same one. This does mean that string equality is slower\nthan equality on other types since it has to walk the whole string. We’ll revise\nthat later, but this gives us the right semantics for now.

\n

Finally, in order to use memcmp() and the new stuff in the “object” module, we\nneed a couple of includes. Here:

\n
#include <stdio.h>\n
value.c
\n
#include <string.h>\n
\n\n#include "memory.h"\n
\n
value.c
\n\n

And here:

\n
#include <string.h>\n\n
value.c
\n
#include "object.h"\n
#include "memory.h"\n
\n
value.c
\n\n

19 . 4 . 1Concatenation

\n

Full-grown languages provide lots of operations for working with strings—access to individual characters, the string’s length, changing case, splitting,\njoining, searching, etc. When you implement your language, you’ll likely want\nall that. But for this book, we keep things very minimal.

\n

The only interesting operation we support on strings is +. If you use that\noperator on two string objects, it produces a new string that’s a concatenation\nof the two operands. Since Lox is dynamically typed, we can’t tell which\nbehavior is needed at compile time because we don’t know the types of the\noperands until runtime. Thus, the OP_ADD instruction dynamically inspects the\noperands and chooses the right operation.

\n
      case OP_LESS:     BINARY_OP(BOOL_VAL, <); break;\n
vm.c
\nin run()
\nreplace 1 line
\n
      case OP_ADD: {\n        if (IS_STRING(peek(0)) && IS_STRING(peek(1))) {\n          concatenate();\n        } else if (IS_NUMBER(peek(0)) && IS_NUMBER(peek(1))) {\n          double b = AS_NUMBER(pop());\n          double a = AS_NUMBER(pop());\n          push(NUMBER_VAL(a + b));\n        } else {\n          runtimeError(\n              "Operands must be two numbers or two strings.");\n          return INTERPRET_RUNTIME_ERROR;\n        }\n        break;\n      }\n
      case OP_SUBTRACT: BINARY_OP(NUMBER_VAL, -); break;\n
\n
vm.c, in run(), replace 1 line
\n\n

If both operands are strings, it concatenates. If they’re both numbers, it adds\nthem. Any other combination of operand types is a\nruntime error.

\n\n

To concatenate strings, we define a new function.

\n
vm.c
\nadd after isFalsey()
\n
static void concatenate() {\n  ObjString* b = AS_STRING(pop());\n  ObjString* a = AS_STRING(pop());\n\n  int length = a->length + b->length;\n  char* chars = ALLOCATE(char, length + 1);\n  memcpy(chars, a->chars, a->length);\n  memcpy(chars + a->length, b->chars, b->length);\n  chars[length] = '\\0';\n\n  ObjString* result = takeString(chars, length);\n  push(OBJ_VAL(result));\n}\n
\n
vm.c, add after isFalsey()
\n\n

It’s pretty verbose, as C code that works with strings tends to be. First, we\ncalculate the length of the result string based on the lengths of the operands.\nWe allocate a character array for the result and then copy the two halves in. As\nalways, we carefully ensure the string is terminated.

\n

In order to call memcpy(), the VM needs an include.

\n
#include <stdio.h>\n
vm.c
\n
#include <string.h>\n
\n\n#include "common.h"\n
\n
vm.c
\n\n

Finally, we produce an ObjString to contain those characters. This time we use a\nnew function, takeString().

\n
};\n\n
object.h
\nadd after struct ObjString
\n
ObjString* takeString(char* chars, int length);\n
ObjString* copyString(const char* chars, int length);\n
\n
object.h, add after struct ObjString
\n\n

The implementation looks like this:

\n
object.c
\nadd after allocateString()
\n
ObjString* takeString(char* chars, int length) {\n  return allocateString(chars, length);\n}\n
\n
object.c, add after allocateString()
\n\n

The previous copyString() function assumes it cannot take ownership of the\ncharacters you pass in. Instead, it conservatively creates a copy of the\ncharacters on the heap that the ObjString can own. That’s the right thing for\nstring literals where the passed-in characters are in the middle of the source\nstring.

\n

But, for concatenation, we’ve already dynamically allocated a character array on\nthe heap. Making another copy of that would be redundant (and would mean\nconcatenate() has to remember to free its copy). Instead, this function claims\nownership of the string you give it.

\n

As usual, stitching this functionality together requires a couple of includes.

\n
#include "debug.h"\n
vm.c
\n
#include "object.h"\n#include "memory.h"\n
#include "vm.h"\n
\n
vm.c
\n\n

19 . 5Freeing Objects

\n

Behold this innocuous-seeming expression:

\n
"st" + "ri" + "ng"\n
\n

When the compiler chews through this, it allocates an ObjString for each of\nthose three string literals and stores them in the chunk’s constant table and\ngenerates this bytecode:

\n\n
0000    OP_CONSTANT         0 "st"\n0002    OP_CONSTANT         1 "ri"\n0004    OP_ADD\n0005    OP_CONSTANT         2 "ng"\n0007    OP_ADD\n0008    OP_RETURN\n
\n

The first two instructions push \"st\" and \"ri\" onto the stack. Then the\nOP_ADD pops those and concatenates them. That dynamically allocates a new\n\"stri\" string on the heap. The VM pushes that and then pushes the \"ng\"\nconstant. The last OP_ADD pops \"stri\" and \"ng\", concatenates them, and\npushes the result: \"string\". Great, that’s what we expect.

\n

But, wait. What happened to that \"stri\" string? We dynamically allocated it,\nthen the VM discarded it after concatenating it with \"ng\". We popped it from\nthe stack and no longer have a reference to it, but we never freed its memory.\nWe’ve got ourselves a classic memory leak.

\n

Of course, it’s perfectly fine for the Lox program to forget about\nintermediate strings and not worry about freeing them. Lox automatically manages\nmemory on the user’s behalf. The responsibility to manage memory doesn’t\ndisappear. Instead, it falls on our shoulders as VM implementers.

\n

The full solution is a garbage collector that\nreclaims unused memory while the program is running. We’ve got some other stuff\nto get in place before we’re ready to tackle that project. Until then, we are\nliving on borrowed time. The longer we wait to add the collector, the harder it\nis to do.

\n\n

Today, we should at least do the bare minimum: avoid leaking memory by making\nsure the VM can still find every allocated object even if the Lox program itself\nno longer references them. There are many sophisticated techniques that advanced\nmemory managers use to allocate and track memory for objects. We’re going to\ntake the simplest practical approach.

\n

We’ll create a linked list that stores every Obj. The VM can traverse that\nlist to find every single object that has been allocated on the heap, whether or\nnot the user’s program or the VM’s stack still has a reference to it.

\n

We could define a separate linked list node struct but then we’d have to\nallocate those too. Instead, we’ll use an intrusive list—the Obj struct\nitself will be the linked list node. Each Obj gets a pointer to the next Obj in\nthe chain.

\n
struct Obj {\n  ObjType type;\n
object.h
\nin struct Obj
\n
  struct Obj* next;\n
};\n
\n
object.h, in struct Obj
\n\n

The VM stores a pointer to the head of the list.

\n
  Value* stackTop;\n
vm.h
\nin struct VM
\n
  Obj* objects;\n
} VM;\n
\n
vm.h, in struct VM
\n\n

When we first initialize the VM, there are no allocated objects.

\n
  resetStack();\n
vm.c
\nin initVM()
\n
  vm.objects = NULL;\n
}\n
\n
vm.c, in initVM()
\n\n

Every time we allocate an Obj, we insert it in the list.

\n
  object->type = type;\n
object.c
\nin allocateObject()
\n
\n\n  object->next = vm.objects;\n  vm.objects = object;\n
  return object;\n
\n
object.c, in allocateObject()
\n\n

Since this is a singly linked list, the easiest place to insert it is as the\nhead. That way, we don’t need to also store a pointer to the tail and keep it\nupdated.

\n

The “object” module is directly using the global vm variable from the “vm”\nmodule, so we need to expose that externally.

\n
} InterpretResult;\n\n
vm.h
\nadd after enum InterpretResult
\n
extern VM vm;\n\n
void initVM();\n
\n
vm.h, add after enum InterpretResult
\n\n

Eventually, the garbage collector will free memory while the VM is still\nrunning. But, even then, there will usually be unused objects still lingering in\nmemory when the user’s program completes. The VM should free those too.

\n

There’s no sophisticated logic for that. Once the program is done, we can free\nevery object. We can and should implement that now.

\n
void freeVM() {\n
vm.c
\nin freeVM()
\n
  freeObjects();\n
}\n
\n
vm.c, in freeVM()
\n\n

That empty function we defined way back when finally does something! It\ncalls this:

\n
void* reallocate(void* pointer, size_t oldSize, size_t newSize);\n
memory.h
\nadd after reallocate()
\n
void freeObjects();\n
\n\n#endif\n
\n
memory.h, add after reallocate()
\n\n

Here’s how we free the objects:

\n
memory.c
\nadd after reallocate()
\n
void freeObjects() {\n  Obj* object = vm.objects;\n  while (object != NULL) {\n    Obj* next = object->next;\n    freeObject(object);\n    object = next;\n  }\n}\n
\n
memory.c, add after reallocate()
\n\n

This is a CS 101 textbook implementation of walking a linked list and freeing\nits nodes. For each node, we call:

\n
memory.c
\nadd after reallocate()
\n
static void freeObject(Obj* object) {\n  switch (object->type) {\n    case OBJ_STRING: {\n      ObjString* string = (ObjString*)object;\n      FREE_ARRAY(char, string->chars, string->length + 1);\n      FREE(ObjString, object);\n      break;\n    }\n  }\n}\n
\n
memory.c, add after reallocate()
\n\n

We aren’t only freeing the Obj itself. Since some object types also allocate\nother memory that they own, we also need a little type-specific code to handle\neach object type’s special needs. Here, that means we free the character array\nand then free the ObjString. Those both use one last memory management macro.

\n
    (type*)reallocate(NULL, 0, sizeof(type) * (count))\n
memory.h
\n
\n\n#define FREE(type, pointer) reallocate(pointer, sizeof(type), 0)\n
\n\n#define GROW_CAPACITY(capacity) \\\n
\n
memory.h
\n\n

It’s a tiny wrapper around reallocate() that\n“resizes” an allocation down to zero bytes.

\n\n

As usual, we need an include to wire everything together.

\n
#include "common.h"\n
memory.h
\n
#include "object.h"\n
\n\n#define ALLOCATE(type, count) \\\n
\n
memory.h
\n\n

Then in the implementation file:

\n
#include "memory.h"\n
memory.c
\n
#include "vm.h"\n
\n\nvoid* reallocate(void* pointer, size_t oldSize, size_t newSize) {\n
\n
memory.c
\n\n

With this, our VM no longer leaks memory. Like a good C program, it cleans up\nits mess before exiting. But it doesn’t free any objects while the VM is\nrunning. Later, when it’s possible to write longer-running Lox programs, the VM\nwill eat more and more memory as it goes, not relinquishing a single byte until\nthe entire program is done.

\n

We won’t address that until we’ve added a real garbage collector, but this\nis a big step. We now have the infrastructure to support a variety of different\nkinds of dynamically allocated objects. And we’ve used that to add strings to\nclox, one of the most used types in most programming languages. Strings in turn\nenable us to build another fundamental data type, especially in dynamic\nlanguages: the venerable hash table. But that’s for the next chapter . . . 

\n
\n

Challenges

\n
    \n
  1. \n

    Each string requires two separate dynamic allocations—one for the\nObjString and a second for the character array. Accessing the characters\nfrom a value requires two pointer indirections, which can be bad for\nperformance. A more efficient solution relies on a technique called\nflexible array members. Use that to store the ObjString and its\ncharacter array in a single contiguous allocation.

    \n
    2. \n
  2. \n

    When we create the ObjString for each string literal, we copy the characters\nonto the heap. That way, when the string is later freed, we know it is safe\nto free the characters too.

    \n

    This is a simpler approach but wastes some memory, which might be a problem\non very constrained devices. Instead, we could keep track of which\nObjStrings own their character array and which are “constant strings” that\njust point back to the original source string or some other non-freeable\nlocation. Add support for this.

    \n
    3. \n
  3. \n

    If Lox was your language, what would you have it do when a user tries to use\n+ with one string operand and the other some other type? Justify your\nchoice. What do other languages do?

    \n
    4. \n
\n
\n
\n

Design Note: String Encoding

\n

In this book, I try not to shy away from the gnarly problems you’ll run into in\na real language implementation. We might not always use the most sophisticated\nsolution—it’s an intro book after all—but I don’t think it’s honest to\npretend the problem doesn’t exist at all. However, I did skirt around one really\nnasty conundrum: deciding how to represent strings.

\n

There are two facets to a string encoding:

\n
    \n
  • \n

    What is a single “character” in a string? How many different values are\nthere and what do they represent? The first widely adopted standard answer\nto this was ASCII. It gave you 127 different character values and\nspecified what they were. It was great . . . if you only ever cared about\nEnglish. While it has weird, mostly forgotten characters like “record\nseparator” and “synchronous idle”, it doesn’t have a single umlaut, acute,\nor grave. It can’t represent “jalapeño”, “naïve”, “Gruyère”, or “Mötley Crüe”.

    \n\n

    Next came Unicode. Initially, it supported 16,384 different characters\n(code points), which fit nicely in 16 bits with a couple of bits to\nspare. Later that grew and grew, and now there are well over 100,000\ndifferent code points including such vital instruments of human\ncommunication as 💩 (Unicode Character ‘PILE OF POO’, U+1F4A9).

    \n

    Even that long list of code points is not enough to represent each possible\nvisible glyph a language might support. To handle that, Unicode also has\ncombining characters that modify a preceding code point. For example,\n“a” followed by the combining character “¨” gives you “ä”. (To make things\nmore confusing Unicode also has a single code point that looks like “ä”.)

    \n

    If a user accesses the fourth “character” in “naïve”, do they expect to get\nback “v” or “¨”? The former means they are thinking of each code\npoint and its combining character as a single unit—what Unicode calls an\nextended grapheme cluster—the latter means they are thinking in\nindividual code points. Which do your users expect?

    \n
    • \n
  • \n

    How is a single unit represented in memory? Most systems using ASCII\ngave a single byte to each character and left the high bit unused. Unicode\nhas a handful of common encodings. UTF-16 packs most code points into 16\nbits. That was great when every code point fit in that size. When that\noverflowed, they added surrogate pairs that use multiple 16-bit code units\nto represent a single code point. UTF-32 is the next evolution of\nUTF-16—it gives a full 32 bits to each and every code point.

    \n

    UTF-8 is more complex than either of those. It uses a variable number of\nbytes to encode a code point. Lower-valued code points fit in fewer bytes.\nSince each character may occupy a different number of bytes, you can’t\ndirectly index into the string to find a specific code point. If you want,\nsay, the 10th code point, you don’t know how many bytes into the string that\nis without walking and decoding all of the preceding ones.

    \n
    • \n
\n

Choosing a character representation and encoding involves fundamental\ntrade-offs. Like many things in engineering, there’s no perfect solution:

\n\n
    \n
  • \n

    ASCII is memory efficient and fast, but it kicks non-Latin languages to the\nside.

    \n
    • \n
  • \n

    UTF-32 is fast and supports the whole Unicode range, but wastes a lot of\nmemory given that most code points do tend to be in the lower range of\nvalues, where a full 32 bits aren’t needed.

    \n
    • \n
  • \n

    UTF-8 is memory efficient and supports the whole Unicode range, but its\nvariable-length encoding makes it slow to access arbitrary code points.

    \n
    • \n
  • \n

    UTF-16 is worse than all of them—an ugly consequence of Unicode\noutgrowing its earlier 16-bit range. It’s less memory efficient than UTF-8\nbut is still a variable-length encoding thanks to surrogate pairs. Avoid it\nif you can. Alas, if your language needs to run on or interoperate with the\nbrowser, the JVM, or the CLR, you might be stuck with it, since those all\nuse UTF-16 for their strings and you don’t want to have to convert every\ntime you pass a string to the underlying system.

    \n
    • \n
\n

One option is to take the maximal approach and do the “rightest” thing. Support\nall the Unicode code points. Internally, select an encoding for each string\nbased on its contents—use ASCII if every code point fits in a byte, UTF-16 if\nthere are no surrogate pairs, etc. Provide APIs to let users iterate over both\ncode points and extended grapheme clusters.

\n

This covers all your bases but is really complex. It’s a lot to implement,\ndebug, and optimize. When serializing strings or interoperating with other\nsystems, you have to deal with all of the encodings. Users need to understand\nthe two indexing APIs and know which to use when. This is the approach that\nnewer, big languages tend to take—like Raku and Swift.

\n

A simpler compromise is to always encode using UTF-8 and only expose an API that\nworks with code points. For users that want to work with grapheme clusters, let\nthem use a third-party library for that. This is less Latin-centric than ASCII\nbut not much more complex. You lose fast direct indexing by code point, but you\ncan usually live without that or afford to make it O(n) instead of O(1).

\n

If I were designing a big workhorse language for people writing large\napplications, I’d probably go with the maximal approach. For my little embedded\nscripting language Wren, I went with UTF-8 and code points.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/style.css", "content": "@charset \"UTF-8\";\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-roman.woff\") format(\"woff\");\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-italic.woff\") format(\"woff\");\n font-style: italic;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-semibold.woff\") format(\"woff\");\n font-weight: 600;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-semibolditalic.woff\") format(\"woff\");\n font-style: italic;\n font-weight: 600;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-bold.woff\") format(\"woff\");\n font-weight: bold;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-bolditalic.woff\") format(\"woff\");\n font-style: italic;\n font-weight: bold;\n}\nbody, h1, h2, h3, h4, p, blockquote, code, ul, ol, dl, dd, img {\n margin: 0;\n}\n\nimg {\n outline: none;\n}\n\nimg.arrow {\n width: auto;\n height: 11px;\n}\n\nimg.dot {\n width: auto;\n height: 18px;\n vertical-align: text-bottom;\n}\n\nbody {\n color: #222;\n font: normal 16px/24px \"Crimson\", Georgia, serif;\n}\n\narticle.chapter h2 {\n font: 600 30px/24px \"Crimson\", Georgia, serif;\n margin: 69px 0 0 0;\n padding-bottom: 3px;\n}\narticle.chapter h2 small {\n font: 800 22px/24px \"Crimson\", Georgia, serif;\n float: right;\n}\narticle.chapter h3 {\n font: italic 24px/24px \"Crimson\", Georgia, serif;\n margin: 71px 0 0 0;\n padding-bottom: 1px;\n}\narticle.chapter h3 small {\n font: 600 16px/24px \"Crimson\", Georgia, serif;\n float: right;\n}\narticle.chapter h2 a, article.chapter h3 a {\n color: #222;\n border-bottom: none;\n}\narticle.chapter h2 a:hover, article.chapter h3 a:hover {\n border-bottom: none;\n color: inherit;\n}\narticle.chapter h2 a::before, article.chapter h3 a::before {\n position: absolute;\n left: -48px;\n width: 48px;\n content: \"§\";\n color: #fff;\n transition: color 0.2s ease;\n text-align: center;\n}\narticle.chapter h2 a:hover::before, article.chapter h3 a:hover::before {\n color: #ddd;\n}\narticle.chapter .challenges, article.chapter .design-note {\n border-radius: 3px;\n padding: 12px;\n margin: -2px -12px 26px -12px;\n font: normal 16px/24px \"Source Sans Pro\", sans-serif;\n color: #444;\n}\narticle.chapter .challenges h2, article.chapter .design-note h2 {\n margin: 0 0 -12px 0;\n padding: 0;\n font: 600 16px/24px \"Source Sans Pro\", sans-serif;\n text-transform: uppercase;\n letter-spacing: 1px;\n}\narticle.chapter .challenges h2 a, article.chapter .design-note h2 a {\n color: inherit;\n}\narticle.chapter .challenges h2 a::before, article.chapter .design-note h2 a::before {\n content: none;\n}\narticle.chapter .challenges ol, article.chapter .design-note ol {\n padding: 0 0 0 18px;\n}\narticle.chapter .challenges ol li, article.chapter .design-note ol li {\n padding: 0 0 0 6px;\n font-weight: 600;\n}\narticle.chapter .challenges ol li p, article.chapter .design-note ol li p {\n font-weight: 400;\n}\narticle.chapter .challenges pre, article.chapter .design-note pre {\n margin: 0;\n}\narticle.chapter .challenges > blockquote p, article.chapter .design-note > blockquote p {\n margin: 0 24px;\n font: italic 16px/24px \"Source Sans Pro\", sans-serif;\n color: #444;\n}\narticle.chapter .challenges > blockquote::before, article.chapter .challenges > blockquote::after, article.chapter .design-note > blockquote::before, article.chapter .design-note > blockquote::after {\n content: none;\n}\narticle.chapter .challenges aside code, article.chapter .challenges aside .codehilite, article.chapter .design-note aside code, article.chapter .design-note aside .codehilite {\n color: #595959;\n background: #faf8f5;\n}\narticle.chapter .challenges *:last-child, article.chapter .design-note *:last-child {\n margin-bottom: 0;\n}\narticle.chapter .challenges .codehilite,\narticle.chapter .design-note .codehilite {\n margin: -12px 0 -12px 0;\n}\narticle.chapter .challenges {\n background: #eef4f7;\n}\narticle.chapter .challenges code, article.chapter .challenges .codehilite {\n background: #e4eef1;\n}\narticle.chapter .design-note {\n background: #f6f8f2;\n}\narticle.chapter .design-note code, article.chapter .design-note .codehilite {\n background: #eef1ea;\n}\narticle.chapter table {\n width: 100%;\n border-collapse: collapse;\n}\narticle.chapter table thead {\n font: 700 15px \"Crimson\", Georgia, serif;\n}\narticle.chapter table td {\n border-bottom: solid 1px #dee9ed;\n line-height: 22px;\n padding: 3px 0 0 0;\n margin: 0;\n}\narticle.chapter table td + td {\n padding-left: 12px;\n}\n\n@media only screen and (max-width: 960px) {\n article.chapter .challenges aside, article.chapter .design-note aside {\n font: normal 15px/24px \"Source Sans Pro\", sans-serif;\n padding-bottom: 4px;\n }\n article.chapter .challenges aside code, article.chapter .challenges aside .codehilite {\n background: #e4eef1;\n }\n article.chapter .design-note aside code, article.chapter .design-note aside .codehilite {\n background: #eef1ea;\n }\n}\n@media only screen and (max-width: 630px) {\n article.chapter h2 a::before, article.chapter h3 a::before {\n left: -24px;\n width: 24px;\n }\n}\n@media only screen and (max-width: 580px) {\n article.chapter h2 {\n margin-top: 64px;\n padding-bottom: 2px;\n font-size: 22px;\n line-height: 22px;\n }\n article.chapter h3 {\n margin-top: 64px;\n padding-bottom: 0;\n font-size: 20px;\n }\n article.chapter .challenges, article.chapter .design-note {\n padding: 11px 11px 8px 11px;\n margin: 25px 0 0 0;\n font-size: 15px;\n line-height: 22px;\n }\n article.chapter .challenges code, article.chapter .challenges .codehilite, article.chapter .design-note code, article.chapter .design-note .codehilite {\n font-size: 14px;\n }\n article.chapter .challenges h2, article.chapter .design-note h2 {\n padding: 5px 0 4px 6px;\n font-size: 17px;\n line-height: 22px;\n }\n article.chapter .challenges aside, article.chapter .design-note aside {\n line-height: 22px;\n }\n}\narticle.contents h2 {\n margin: 22px 0 6px 0;\n font: 600 normal 18px/24px \"Source Sans Pro\", sans-serif;\n text-transform: uppercase;\n letter-spacing: 1px;\n}\narticle.contents h2 .num {\n display: inline-block;\n width: 36px;\n}\narticle.contents ul {\n margin: -12px 0 0 0;\n padding: 6px 0 14px 0;\n}\narticle.contents li {\n padding: 12px 0 0 36px;\n font: normal 16px/24px \"Source Sans Pro\", sans-serif;\n color: #7aa0b8;\n list-style-type: none;\n}\narticle.contents li .num {\n display: inline-block;\n letter-spacing: 1px;\n width: 36px;\n}\narticle.contents li a {\n font: 600 17px/24px \"Source Sans Pro\", sans-serif;\n}\narticle.contents li.design-note {\n padding-top: 0;\n}\narticle.contents li.design-note a {\n font: 400 16px/23px \"Source Sans Pro\", sans-serif;\n}\narticle.contents .chapters {\n display: table;\n width: 864px;\n}\narticle.contents .row {\n display: table-row;\n}\narticle.contents .first, article.contents .second {\n display: table-cell;\n vertical-align: top;\n}\narticle.contents .second {\n padding-left: 48px;\n}\narticle.contents footer {\n width: 864px;\n}\n\n@media only screen and (max-width: 1344px) {\n article.contents .chapters, article.contents .row, article.contents .first, article.contents .second {\n display: block;\n width: auto;\n }\n article.contents .second {\n padding-left: 0;\n }\n article.contents footer {\n width: inherit;\n }\n}\n@media only screen and (max-width: 630px) {\n article.contents h2 .num, article.contents li .num {\n width: 28px;\n }\n article.contents ol, article.contents ul {\n margin-left: 0;\n }\n article.contents li {\n padding-left: 0;\n }\n}\n@media only screen and (max-width: 580px) {\n article.contents h2 {\n margin: 19px 0 6px 0;\n font-size: 17px;\n line-height: 22px;\n }\n article.contents h3 {\n padding: 1px 0 2px 0;\n font-size: 17px;\n line-height: 22px;\n }\n article.contents p {\n font-size: 15px;\n line-height: 22px;\n }\n article.contents ol, article.contents ul {\n padding-bottom: 8px;\n }\n article.contents li {\n font-size: 14px;\n line-height: 22px;\n padding: 4px 0 3px 0;\n }\n}\n.sign-up {\n padding: 12px;\n margin: 24px 0 24px 0;\n background: #fcf6e8;\n color: #bf9540;\n border-radius: 3px;\n}\n.sign-up form {\n display: flex;\n}\n.sign-up input {\n padding: 4px;\n font: 16px \"Source Sans Pro\", sans-serif;\n outline: none;\n border-radius: 3px;\n border: solid 2px #ffd580;\n color: #825e17;\n height: 32px;\n}\n.sign-up input.email {\n display: block;\n box-sizing: border-box;\n width: 100%;\n}\n.sign-up input.button {\n margin-left: 8px;\n padding: 4px 8px;\n font: 600 13px \"Source Sans Pro\", sans-serif;\n text-transform: uppercase;\n letter-spacing: 1px;\n background: #ffbb33;\n border: none;\n transition: background-color 0.2s ease;\n}\n.sign-up input.button:hover {\n background: #ffd580;\n}\n.sign-up input:focus {\n border-color: #ffaa00;\n}\n\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-roman.woff\") format(\"woff\");\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-italic.woff\") format(\"woff\");\n font-style: italic;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-semibold.woff\") format(\"woff\");\n font-weight: 600;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-semibolditalic.woff\") format(\"woff\");\n font-style: italic;\n font-weight: 600;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-bold.woff\") format(\"woff\");\n font-weight: bold;\n}\n@font-face {\n font-family: \"Crimson\";\n src: url(\"font/crimson-bolditalic.woff\") format(\"woff\");\n font-style: italic;\n font-weight: bold;\n}\nbody, h1, h2, h3, h4, p, blockquote, code, ul, ol, dl, dd, img {\n margin: 0;\n}\n\nimg {\n outline: none;\n}\n\nimg.arrow {\n width: auto;\n height: 11px;\n}\n\nimg.dot {\n width: auto;\n height: 18px;\n vertical-align: text-bottom;\n}\n\nbody {\n color: #222;\n font: normal 16px/24px \"Crimson\", Georgia, serif;\n}\n\n@media print {\n body, a, code {\n color: #000 !important;\n background: none !important;\n }\n\n nav, .sign-up {\n display: none;\n }\n\n .page {\n margin: 0 !important;\n }\n\n .codehilite {\n margin: 0 !important;\n background: none !important;\n border-radius: 0 !important;\n border-left: solid 1px #dad8d6;\n border-right: solid 1px #dad8d6;\n }\n .codehilite pre {\n color: #000 !important;\n }\n .codehilite .insert {\n border-left: solid 3px #dad8d6 !important;\n border-right: solid 3px #dad8d6 !important;\n background: none !important;\n }\n .codehilite .delete {\n -webkit-print-color-adjust: exact;\n color-adjust: exact;\n }\n .codehilite .insert-before span, .codehilite .insert-after span {\n -webkit-print-color-adjust: exact;\n color-adjust: exact;\n }\n}\n.emdash {\n white-space: nowrap;\n}\n\n.scrim {\n position: absolute;\n width: 100%;\n height: 10000px;\n z-index: 4;\n background: url(\"rows.png\");\n}\n\n.small-caps {\n font-weight: 600;\n font-size: 13px;\n}\n\na {\n color: #1481b8;\n text-decoration: none;\n border-bottom: solid 1px rgba(222, 233, 237, 0);\n transition: color 0.2s ease, border-color 0.4s ease;\n}\n\na:hover {\n color: #1481b8;\n border-bottom: solid 1px #dee9ed;\n}\n\nnav {\n font: 300 15px/24px \"Source Sans Pro\", sans-serif;\n background: #29313d;\n color: #4b6781;\n}\nnav a, nav h2 a {\n color: #7aa0b8;\n text-decoration: none;\n border-bottom: none;\n}\nnav a:hover {\n color: #dee9ed;\n text-decoration: none;\n border-bottom: none;\n}\nnav img {\n box-sizing: border-box;\n width: 100%;\n padding: 55px 48px 23px 48px;\n}\nnav h2 {\n font: 400 16px/24px \"Source Sans Pro\", sans-serif;\n text-transform: uppercase;\n letter-spacing: 1px;\n color: #7aa0b8;\n}\nnav h3 {\n font: 400 18px/24px \"Source Sans Pro\", sans-serif;\n color: #7aa0b8;\n}\nnav h2 small, nav h3 small {\n float: right;\n font-size: 16px;\n color: #4b6781;\n}\nnav ol, nav ul {\n margin: 6px 0 3px 0;\n padding: 6px 0 4px 24px;\n border-top: solid 1px #3b4b5e;\n border-bottom: solid 1px #3b4b5e;\n}\nnav ul {\n list-style-type: none;\n padding-left: 0;\n}\nnav hr {\n border: none;\n border-top: solid 1px #3b4b5e;\n margin: 6px 0 0 0;\n padding: 0 0 3px 0;\n}\nnav li small {\n float: right;\n font-size: 14px;\n color: #4b6781;\n}\nnav li.divider {\n margin: 5px 0 7px 0;\n border-top: solid 1px #3b4b5e;\n}\nnav li.end-part {\n font-size: 12px;\n font-weight: 400;\n text-transform: uppercase;\n letter-spacing: 1px;\n}\nnav li.end-part small {\n font-weight: 300;\n text-transform: none;\n letter-spacing: 0;\n}\nnav .prev-next {\n padding-top: 7px;\n font: 400 12px/18px \"Source Sans Pro\", sans-serif;\n text-align: center;\n text-transform: uppercase;\n letter-spacing: 1px;\n}\n\nnav.wide {\n position: fixed;\n width: 336px;\n height: 100%;\n}\nnav.wide .contents {\n margin: 24px 48px;\n}\n\n.nav-wrapper {\n position: absolute;\n right: 288px;\n}\n\nnav.floating {\n display: none;\n z-index: 2;\n position: absolute;\n width: 288px;\n border-bottom-left-radius: 3px;\n border-bottom-right-radius: 3px;\n}\nnav.floating #expand-nav {\n padding: 0 0 4px 0;\n display: block;\n font-size: 20px;\n text-align: center;\n color: #4b6781;\n cursor: pointer;\n transition: padding 0.2s ease, margin 0.2s ease, color 0.2s ease;\n}\nnav.floating #expand-nav, nav.floating #expand-nav:hover {\n border-bottom: none;\n}\nnav.floating #expand-nav:hover {\n color: #dee9ed;\n}\nnav.floating .expandable {\n overflow: hidden;\n padding: 0 12px;\n max-height: 0;\n transition: margin 0.2s ease, max-height 1s ease;\n}\nnav.floating .expandable .prev-next {\n padding-bottom: 6px;\n}\nnav.floating .expandable.shown {\n max-height: 550px;\n}\nnav.floating img {\n padding: 110px 24px 23px 24px;\n}\n\nnav.floating.pinned {\n position: fixed;\n top: -85px;\n}\nnav.floating.pinned .expandable {\n margin-top: -13px;\n}\nnav.floating.pinned #expand-nav {\n margin-top: -14px;\n}\n\nnav.narrow {\n display: none;\n text-align: center;\n}\nnav.narrow img {\n box-sizing: content-box;\n padding: 11px 0 3px 0;\n width: auto;\n height: 27px;\n}\nnav.narrow .prev, nav.narrow .next {\n font-size: 32px;\n position: absolute;\n top: 12px;\n padding: 0 48px;\n}\nnav.narrow .prev {\n left: 0;\n}\nnav.narrow .next {\n right: 0;\n}\n\n.left {\n float: left;\n}\n\n.right {\n float: right;\n}\n\n.page {\n position: relative;\n width: 912px;\n margin: 0 auto 0 384px;\n}\n\n.em {\n padding: 0 0.1em;\n white-space: nowrap;\n}\n\n.ellipse {\n white-space: nowrap;\n}\n\ncode {\n font: normal 16px \"Source Code Pro\", Menlo, Consolas, Monaco, monospace;\n color: #717170;\n white-space: nowrap;\n padding: 2px;\n}\n\nstrong code {\n font-weight: bold;\n color: inherit;\n}\n\na code {\n color: #1481b8;\n}\n\n.codehilite {\n color: #595959;\n background: #faf8f5;\n border-radius: 3px;\n padding: 12px;\n margin: -12px;\n}\n\npre {\n font: normal 13px/20px \"Source Code Pro\", Menlo, Consolas, Monaco, monospace;\n margin: 0;\n padding: 0;\n white-space: pre-wrap;\n overflow-wrap: anywhere;\n}\n\ndiv.codehilite + div.challenges {\n margin-top: 24px;\n}\n\narticle {\n position: relative;\n width: 576px;\n}\narticle h1 {\n position: relative;\n font: 48px/48px \"Crimson\", Georgia, serif;\n padding: 109px 0 19px 0;\n z-index: 2;\n}\narticle h1.part {\n font: 600 36px/48px \"Source Sans Pro\", sans-serif;\n padding: 108px 0 20px 0;\n text-transform: uppercase;\n letter-spacing: 1px;\n}\narticle .number {\n position: absolute;\n top: 50px;\n left: 624px;\n z-index: 1;\n font: 300 96px \"Source Sans Pro\", sans-serif;\n color: #dee9ed;\n}\narticle p {\n margin: 24px 0;\n}\narticle ol, article ul {\n margin: 24px 0;\n padding: 0 0 0 24px;\n}\narticle img {\n max-width: 100%;\n}\narticle img.wide {\n max-width: none;\n width: 912px;\n}\n\naside {\n position: absolute;\n right: -336px;\n width: 288px;\n font: normal 14px/20px \"Crimson\", Georgia, serif;\n border-top: solid 1px #dee9ed;\n}\naside p {\n margin: 20px 0;\n}\naside p:first-child,\naside img:first-child {\n margin-top: 4px;\n}\naside p:last-child {\n margin-bottom: 4px;\n}\naside code {\n font-size: 14px;\n border-radius: 2px;\n padding: 1px 2px;\n}\naside .codehilite {\n padding: 6px;\n margin: -12px 0;\n}\naside .codehilite:last-child {\n margin-bottom: 4px;\n}\naside img.above {\n position: absolute;\n bottom: 100%;\n margin-bottom: 16px;\n}\naside blockquote {\n margin: 20px 0;\n}\naside blockquote::before, aside blockquote::after {\n content: none;\n}\naside blockquote p {\n margin: 0 12px;\n font: italic 15px/20px \"Crimson\", Georgia, serif;\n color: inherit;\n}\n\naside.bottom {\n border-top: none;\n border-bottom: solid 1px #dee9ed;\n}\n\nblockquote {\n position: relative;\n margin: 29px 0 31px 0;\n}\nblockquote::before, blockquote::after {\n position: absolute;\n top: -20px;\n font: italic 72px \"Crimson\", Georgia, serif;\n color: #dee9ed;\n}\nblockquote::before {\n content: \"“\";\n left: -7px;\n}\nblockquote::after {\n content: \"”\";\n right: 8px;\n}\nblockquote p {\n margin: 0 48px;\n font: italic 24px/36px \"Crimson\", Georgia, serif;\n color: #5985a6;\n}\nblockquote p em {\n font-style: normal;\n}\nblockquote cite {\n display: block;\n text-align: right;\n color: #7aa0b8;\n font-style: normal;\n font-size: 18px;\n}\nblockquote cite::before {\n content: \"— \";\n color: #dee9ed;\n}\nblockquote cite em {\n font-style: italic;\n}\n\nfooter {\n position: relative;\n border-top: solid 1px #dee9ed;\n color: #7aa0b8;\n font: 400 15px \"Source Sans Pro\", sans-serif;\n text-align: center;\n margin: 48px 0;\n padding-top: 48px;\n}\nfooter a, footer a:hover {\n border: none;\n}\nfooter .next {\n position: absolute;\n right: 0;\n top: -13px;\n padding-left: 4px;\n background: #fff;\n font: 400 17px/24px \"Source Sans Pro\", sans-serif;\n text-transform: uppercase;\n letter-spacing: 1px;\n}\nfooter .next:hover {\n color: #004466;\n border: none;\n}\n\n.dedication {\n margin: 96px 0 128px 0;\n text-align: center;\n}\n.dedication img {\n width: 50%;\n}\n\n.source-file, .source-file-narrow {\n font: normal 11px/16px \"Source Code Pro\", Menlo, Consolas, Monaco, monospace;\n color: #bab8b7;\n}\n.source-file em, .source-file-narrow em {\n color: #999997;\n font-style: normal;\n}\n\n.source-file-narrow {\n display: none;\n margin: 0px -12px 0 0;\n padding: 14px 0 0 0;\n text-align: right;\n}\n\n.source-file {\n position: absolute;\n right: -336px;\n width: 288px;\n padding: 2px 0 0 0;\n}\n.source-file::before {\n content: \"<<\";\n color: #dad8d6;\n position: absolute;\n left: -36px;\n width: 36px;\n text-align: center;\n}\n\n.codehilite pre {\n color: #797978;\n}\n.codehilite .k {\n color: #0099e6;\n}\n.codehilite .n {\n color: #dd713c;\n}\n.codehilite .s {\n color: #c38e22;\n}\n.codehilite .e {\n color: #e8ba30;\n}\n.codehilite .c {\n color: #aaa9a7;\n}\n.codehilite .a {\n color: #9966cc;\n}\n.codehilite .i {\n color: #1b6e98;\n}\n.codehilite .t {\n color: #00a4b3;\n}\n.codehilite .insert {\n margin: -2px -12px;\n padding: 2px 10px;\n border-left: solid 2px #dad8d6;\n border-right: solid 2px #dad8d6;\n background: #f5f3f0;\n}\n.codehilite .delete {\n margin: -2px -12px;\n padding: 2px 10px;\n border-left: solid 2px #dad8d6;\n border-right: solid 2px #dad8d6;\n background: repeating-linear-gradient(-45deg, #dad8d6, #dad8d6 1px, rgba(0, 0, 0, 0) 1px, rgba(0, 0, 0, 0) 6px);\n}\n.codehilite .delete span {\n color: #bab8b7;\n}\n.codehilite .insert-before, .codehilite .insert-after {\n color: #bab8b7;\n}\n.codehilite .insert-before .insert-comma {\n margin: -2px -1px;\n padding: 2px 1px;\n border-radius: 2px;\n background: #f5f3f0;\n color: #595959;\n}\n\n@media only screen and (max-width: 1344px) {\n nav.wide {\n display: none;\n }\n\n nav.floating {\n display: block;\n }\n\n body {\n margin: 0 24px;\n }\n\n .page {\n position: relative;\n width: inherit;\n max-width: 912px;\n margin: 0 auto;\n }\n\n article {\n width: inherit;\n margin-right: 336px;\n }\n article .number {\n top: 73px;\n left: inherit;\n right: 0;\n font-size: 72px;\n }\n article h1 {\n padding: 110px 0 18px 0;\n font-size: 44px;\n }\n}\n@media only screen and (max-width: 960px) {\n body {\n margin: 0;\n }\n\n nav.floating {\n display: none;\n }\n\n nav.narrow {\n display: block;\n }\n\n .page {\n margin: 0 48px;\n width: inherit;\n }\n\n article {\n margin: 0;\n }\n article img.wide {\n width: inherit;\n max-width: 100%;\n }\n\n aside {\n position: inherit;\n right: inherit;\n width: inherit;\n border-bottom: solid 1px #dee9ed;\n }\n aside p:first-child {\n margin-top: 8px;\n }\n aside p:last-child {\n margin-bottom: 8px;\n }\n aside div.codehilite:last-child {\n margin-bottom: 12px;\n }\n aside img {\n display: block;\n max-width: 288px;\n margin: 0 auto;\n }\n aside img.above {\n position: relative;\n }\n\n aside + div.codehilite {\n margin-top: 12px;\n }\n\n div.codehilite + aside {\n margin-top: 24px;\n }\n\n .source-file {\n display: none;\n }\n\n .source-file-narrow {\n display: block;\n }\n}\n@media only screen and (max-width: 630px) {\n .page {\n margin: 0 24px;\n width: inherit;\n }\n\n nav.narrow .prev, nav.narrow .next {\n padding: 0 24px;\n }\n}\n@media only screen and (max-width: 580px) {\n body {\n font-size: 15px;\n line-height: 22px;\n }\n\n .small-caps {\n font-size: 12px;\n }\n\n .scrim {\n background: url(\"rows-22.png\");\n }\n\n nav.narrow img {\n padding: 9px 0 1px 0;\n height: 27px;\n }\n nav.narrow .prev, nav.narrow .next {\n top: 11px;\n }\n\n article h1 {\n font-size: 36px;\n padding: 100px 0 14px 0;\n }\n article h1.part {\n font-size: 30px;\n padding: 97px 0 17px 0;\n }\n article .number {\n top: 61px;\n font-size: 72px;\n }\n article p {\n margin: 22px 0;\n }\n article ol, article ul {\n margin: 22px 0;\n padding: 0 0 0 22px;\n }\n\n blockquote {\n margin: 27px 0 28px 0;\n }\n blockquote::before, blockquote::after {\n top: -17px;\n font-size: 52px;\n }\n blockquote p {\n margin: 0 22px;\n font-size: 20px;\n line-height: 33px;\n }\n\n footer .next {\n font-size: 15px;\n }\n}" }, { "path": "site/superclasses.html", "content": "\n\n\n\nSuperclasses · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
29
\n

Superclasses

\n\n
\n

You can choose your friends but you sho’ can’t choose your family, an’ they’re\nstill kin to you no matter whether you acknowledge ’em or not, and it\nmakes you look right silly when you don’t.

\n

Harper Lee, To Kill a Mockingbird

\n
\n

This is the very last chapter where we add new functionality to our VM. We’ve\npacked almost the entire Lox language in there already. All that remains is\ninheriting methods and calling superclass methods. We have another\nchapter after this one, but it introduces no new behavior. It\nonly makes existing stuff faster. Make it to the end\nof this one, and you’ll have a complete Lox implementation.

\n\n

Some of the material in this chapter will remind you of jlox. The way we resolve\nsuper calls is pretty much the same, though viewed through clox’s more complex\nmechanism for storing state on the stack. But we have an entirely different,\nmuch faster, way of handling inherited method calls this time around.

\n

29 . 1Inheriting Methods

\n

We’ll kick things off with method inheritance since it’s the simpler piece. To\nrefresh your memory, Lox inheritance syntax looks like this:

\n
class Doughnut {\n  cook() {\n    print "Dunk in the fryer.";\n  }\n}\n\nclass Cruller < Doughnut {\n  finish() {\n    print "Glaze with icing.";\n  }\n}\n
\n

Here, the Cruller class inherits from Doughnut and thus, instances of Cruller\ninherit the cook() method. I don’t know why I’m belaboring this. You know how\ninheritance works. Let’s start compiling the new syntax.

\n
  currentClass = &classCompiler;\n\n
compiler.c
\nin classDeclaration()
\n
  if (match(TOKEN_LESS)) {\n    consume(TOKEN_IDENTIFIER, "Expect superclass name.");\n    variable(false);\n    namedVariable(className, false);\n    emitByte(OP_INHERIT);\n  }\n\n
  namedVariable(className, false);\n
\n
compiler.c, in classDeclaration()
\n\n

After we compile the class name, if the next token is a <, then we found a\nsuperclass clause. We consume the superclass’s identifier token, then call\nvariable(). That function takes the previously consumed token, treats it as a\nvariable reference, and emits code to load the variable’s value. In other words,\nit looks up the superclass by name and pushes it onto the stack.

\n

After that, we call namedVariable() to load the subclass doing the inheriting\nonto the stack, followed by an OP_INHERIT instruction. That instruction\nwires up the superclass to the new subclass. In the last chapter, we defined an\nOP_METHOD instruction to mutate an existing class object by adding a method to\nits method table. This is similar—the OP_INHERIT instruction takes an\nexisting class and applies the effect of inheritance to it.

\n

In the previous example, when the compiler works through this bit of syntax:

\n
class Cruller < Doughnut {\n
\n

The result is this bytecode:

\"The\n

Before we implement the new OP_INHERIT instruction, we have an edge case to\ndetect.

\n
    variable(false);\n
compiler.c
\nin classDeclaration()
\n
\n\n    if (identifiersEqual(&className, &parser.previous)) {\n      error("A class can't inherit from itself.");\n    }\n\n
    namedVariable(className, false);\n
\n
compiler.c, in classDeclaration()
\n\n

A class cannot be its own superclass. Unless you have\naccess to a deranged nuclear physicist and a very heavily modified DeLorean, you\ncannot inherit from yourself.

\n\n

29 . 1 . 1Executing inheritance

\n

Now onto the new instruction.

\n
  OP_CLASS,\n
chunk.h
\nin enum OpCode
\n
  OP_INHERIT,\n
  OP_METHOD\n
\n
chunk.h, in enum OpCode
\n\n

There are no operands to worry about. The two values we need—superclass and\nsubclass—are both found on the stack. That means disassembling is easy.

\n
      return constantInstruction("OP_CLASS", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_INHERIT:\n      return simpleInstruction("OP_INHERIT", offset);\n
    case OP_METHOD:\n
\n
debug.c, in disassembleInstruction()
\n\n

The interpreter is where the action happens.

\n
        break;\n
vm.c
\nin run()
\n
      case OP_INHERIT: {\n        Value superclass = peek(1);\n        ObjClass* subclass = AS_CLASS(peek(0));\n        tableAddAll(&AS_CLASS(superclass)->methods,\n                    &subclass->methods);\n        pop(); // Subclass.\n        break;\n      }\n
      case OP_METHOD:\n
\n
vm.c, in run()
\n\n

From the top of the stack down, we have the subclass then the superclass. We\ngrab both of those and then do the inherit-y bit. This is where clox takes a\ndifferent path than jlox. In our first interpreter, each subclass stored a\nreference to its superclass. On method access, if we didn’t find the method in\nthe subclass’s method table, we recursed through the inheritance chain looking\nat each ancestor’s method table until we found it.

\n

For example, calling cook() on an instance of Cruller sends jlox on this\njourney:

\"Resolving\n

That’s a lot of work to perform during method invocation time. It’s slow, and\nworse, the farther an inherited method is up the ancestor chain, the slower it\ngets. Not a great performance story.

\n

The new approach is much faster. When the subclass is declared, we copy all of\nthe inherited class’s methods down into the subclass’s own method table. Later,\nwhen calling a method, any method inherited from a superclass will be found\nright in the subclass’s own method table. There is no extra runtime work needed\nfor inheritance at all. By the time the class is declared, the work is done.\nThis means inherited method calls are exactly as fast as normal method calls—a single hash table lookup.

\"Resolving\n\n

I’ve sometimes heard this technique called “copy-down inheritance”. It’s simple\nand fast, but, like most optimizations, you get to use it only under certain\nconstraints. It works in Lox because Lox classes are closed. Once a class\ndeclaration is finished executing, the set of methods for that class can never\nchange.

\n

In languages like Ruby, Python, and JavaScript, it’s possible to crack open an existing class and jam some new methods into\nit or even remove them. That would break our optimization because if those\nmodifications happened to a superclass after the subclass declaration\nexecuted, the subclass would not pick up those changes. That breaks a user’s\nexpectation that inheritance always reflects the current state of the\nsuperclass.

\n\n

Fortunately for us (but not for users who like the feature, I guess), Lox\ndoesn’t let you patch monkeys or punch ducks, so we can safely apply this\noptimization.

\n

What about method overrides? Won’t copying the superclass’s methods into the\nsubclass’s method table clash with the subclass’s own methods? Fortunately, no.\nWe emit the OP_INHERIT after the OP_CLASS instruction that creates the\nsubclass but before any method declarations and OP_METHOD instructions have\nbeen compiled. At the point that we copy the superclass’s methods down, the\nsubclass’s method table is empty. Any methods the subclass overrides will\noverwrite those inherited entries in the table.

\n

29 . 1 . 2Invalid superclasses

\n

Our implementation is simple and fast, which is just the way I like my VM code.\nBut it’s not robust. Nothing prevents a user from inheriting from an object that\nisn’t a class at all:

\n
var NotClass = "So not a class";\nclass OhNo < NotClass {}\n
\n

Obviously, no self-respecting programmer would write that, but we have to guard\nagainst potential Lox users who have no self respect. A simple runtime check\nfixes that.

\n
        Value superclass = peek(1);\n
vm.c
\nin run()
\n
        if (!IS_CLASS(superclass)) {\n          runtimeError("Superclass must be a class.");\n          return INTERPRET_RUNTIME_ERROR;\n        }\n\n
        ObjClass* subclass = AS_CLASS(peek(0));\n
\n
vm.c, in run()
\n\n

If the value we loaded from the identifier in the superclass clause isn’t an\nObjClass, we report a runtime error to let the user know what we think of them\nand their code.

\n

29 . 2Storing Superclasses

\n

Did you notice that when we added method inheritance, we didn’t actually add any\nreference from a subclass to its superclass? After we copy the inherited methods\nover, we forget the superclass entirely. We don’t need to keep a handle on the\nsuperclass, so we don’t.

\n

That won’t be sufficient to support super calls. Since a subclass may override the superclass method, we need to be able to get\nour hands on superclass method tables. Before we get to that mechanism, I want \nto refresh your memory on how super calls are statically resolved.

\n\n

Back in the halcyon days of jlox, I showed you this tricky example to\nexplain the way super calls are dispatched:

\n
class A {\n  method() {\n    print "A method";\n  }\n}\n\nclass B < A {\n  method() {\n    print "B method";\n  }\n\n  test() {\n    super.method();\n  }\n}\n\nclass C < B {}\n\nC().test();\n
\n

Inside the body of the test() method, this is an instance of C. If super\ncalls were resolved relative to the superclass of the receiver, then we would\nlook in C’s superclass, B. But super calls are resolved relative to the\nsuperclass of the surrounding class where the super call occurs. In this case,\nwe are in B’s test() method, so the superclass is A, and the program should\nprint “A method”.

\n

This means that super calls are not resolved dynamically based on the runtime\ninstance. The superclass used to look up the method is a static—practically\nlexical—property of where the call occurs. When we added inheritance to jlox,\nwe took advantage of that static aspect by storing the superclass in the same\nEnvironment structure we used for all lexical scopes. Almost as if the\ninterpreter saw the above program like this:

\n
class A {\n  method() {\n    print "A method";\n  }\n}\n\nvar Bs_super = A;\nclass B < A {\n  method() {\n    print "B method";\n  }\n\n  test() {\n    runtimeSuperCall(Bs_super, "method");\n  }\n}\n\nvar Cs_super = B;\nclass C < B {}\n\nC().test();\n
\n

Each subclass has a hidden variable storing a reference to its superclass.\nWhenever we need to perform a super call, we access the superclass from that\nvariable and tell the runtime to start looking for methods there.

\n

We’ll take the same path with clox. The difference is that instead of jlox’s\nheap-allocated Environment class, we have the bytecode VM’s value stack and\nupvalue system. The machinery is a little different, but the overall effect is\nthe same.

\n

29 . 2 . 1A superclass local variable

\n

Our compiler already emits code to load the superclass onto the stack. Instead\nof leaving that slot as a temporary, we create a new scope and make it a local\nvariable.

\n
    }\n\n
compiler.c
\nin classDeclaration()
\n
    beginScope();\n    addLocal(syntheticToken("super"));\n    defineVariable(0);\n\n
    namedVariable(className, false);\n    emitByte(OP_INHERIT);\n
\n
compiler.c, in classDeclaration()
\n\n

Creating a new lexical scope ensures that if we declare two classes in the same\nscope, each has a different local slot to store its superclass. Since we always\nname this variable “super”, if we didn’t make a scope for each subclass, the\nvariables would collide.

\n

We name the variable “super” for the same reason we use “this” as the name of\nthe hidden local variable that this expressions resolve to: “super” is a\nreserved word, which guarantees the compiler’s hidden variable won’t collide\nwith a user-defined one.

\n

The difference is that when compiling this expressions, we conveniently have a\ntoken sitting around whose lexeme is “this”. We aren’t so lucky here. Instead,\nwe add a little helper function to create a synthetic token for the given constant string.

\n
compiler.c
\nadd after variable()
\n
static Token syntheticToken(const char* text) {\n  Token token;\n  token.start = text;\n  token.length = (int)strlen(text);\n  return token;\n}\n
\n
compiler.c, add after variable()
\n\n\n

Since we opened a local scope for the superclass variable, we need to close it.

\n
  emitByte(OP_POP);\n
compiler.c
\nin classDeclaration()
\n
\n\n  if (classCompiler.hasSuperclass) {\n    endScope();\n  }\n
\n\n  currentClass = currentClass->enclosing;\n
\n
compiler.c, in classDeclaration()
\n\n

We pop the scope and discard the “super” variable after compiling the class body\nand its methods. That way, the variable is accessible in all of the methods of\nthe subclass. It’s a somewhat pointless optimization, but we create the scope\nonly if there is a superclass clause. Thus we need to close the scope only if\nthere is one.

\n

To track that, we could declare a little local variable in classDeclaration().\nBut soon, other functions in the compiler will need to know whether the\nsurrounding class is a subclass or not. So we may as well give our future selves\na hand and store this fact as a field in the ClassCompiler now.

\n
typedef struct ClassCompiler {\n  struct ClassCompiler* enclosing;\n
compiler.c
\nin struct ClassCompiler
\n
  bool hasSuperclass;\n
} ClassCompiler;\n
\n
compiler.c, in struct ClassCompiler
\n\n

When we first initialize a ClassCompiler, we assume it is not a subclass.

\n
  ClassCompiler classCompiler;\n
compiler.c
\nin classDeclaration()
\n
  classCompiler.hasSuperclass = false;\n
  classCompiler.enclosing = currentClass;\n
\n
compiler.c, in classDeclaration()
\n\n

Then, if we see a superclass clause, we know we are compiling a subclass.

\n
    emitByte(OP_INHERIT);\n
compiler.c
\nin classDeclaration()
\n
    classCompiler.hasSuperclass = true;\n
  }\n
\n
compiler.c, in classDeclaration()
\n\n

This machinery gives us a mechanism at runtime to access the superclass object\nof the surrounding subclass from within any of the subclass’s methods—simply\nemit code to load the variable named “super”. That variable is a local outside\nof the method body, but our existing upvalue support enables the VM to capture\nthat local inside the body of the method or even in functions nested inside that\nmethod.

\n

29 . 3Super Calls

\n

With that runtime support in place, we are ready to implement super calls. As\nusual, we go front to back, starting with the new syntax. A super call begins, naturally enough, with the super keyword.

\n\n
  [TOKEN_RETURN]        = {NULL,     NULL,   PREC_NONE},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_SUPER]         = {super_,   NULL,   PREC_NONE},\n
  [TOKEN_THIS]          = {this_,    NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n

When the expression parser lands on a super token, control jumps to a new\nparsing function which starts off like so:

\n
compiler.c
\nadd after syntheticToken()
\n
static void super_(bool canAssign) {\n  consume(TOKEN_DOT, "Expect '.' after 'super'.");\n  consume(TOKEN_IDENTIFIER, "Expect superclass method name.");\n  uint8_t name = identifierConstant(&parser.previous);\n}\n
\n
compiler.c, add after syntheticToken()
\n\n

This is pretty different from how we compiled this expressions. Unlike this,\na super token is not a standalone expression.\nInstead, the dot and method name following it are inseparable parts of the\nsyntax. However, the parenthesized argument list is separate. As with normal\nmethod access, Lox supports getting a reference to a superclass method as a\nclosure without invoking it:

\n\n
class A {\n  method() {\n    print "A";\n  }\n}\n\nclass B < A {\n  method() {\n    var closure = super.method;\n    closure(); // Prints "A".\n  }\n}\n
\n

In other words, Lox doesn’t really have super call expressions, it has super\naccess expressions, which you can choose to immediately invoke if you want. So\nwhen the compiler hits a super token, we consume the subsequent . token and\nthen look for a method name. Methods are looked up dynamically, so we use\nidentifierConstant() to take the lexeme of the method name token and store it\nin the constant table just like we do for property access expressions.

\n

Here is what the compiler does after consuming those tokens:

\n
  uint8_t name = identifierConstant(&parser.previous);\n
compiler.c
\nin super_()
\n
\n\n  namedVariable(syntheticToken("this"), false);\n  namedVariable(syntheticToken("super"), false);\n  emitBytes(OP_GET_SUPER, name);\n
}\n
\n
compiler.c, in super_()
\n\n

In order to access a superclass method on the current instance, the runtime\nneeds both the receiver and the superclass of the surrounding method’s class.\nThe first namedVariable() call generates code to look up the current receiver\nstored in the hidden variable “this” and push it onto the stack. The second\nnamedVariable() call emits code to look up the superclass from its “super”\nvariable and push that on top.

\n

Finally, we emit a new OP_GET_SUPER instruction with an operand for the\nconstant table index of the method name. That’s a lot to hold in your head. To\nmake it tangible, consider this example program:

\n
class Doughnut {\n  cook() {\n    print "Dunk in the fryer.";\n    this.finish("sprinkles");\n  }\n\n  finish(ingredient) {\n    print "Finish with " + ingredient;\n  }\n}\n\nclass Cruller < Doughnut {\n  finish(ingredient) {\n    // No sprinkles, always icing.\n    super.finish("icing");\n  }\n}\n
\n

The bytecode emitted for the super.finish(\"icing\") expression looks and works\nlike this:

\"The\n

The first three instructions give the runtime access to the three pieces of\ninformation it needs to perform the super access:

\n
    \n
  1. \n

    The first instruction loads the instance onto the stack.

    \n
    2. \n
  2. \n

    The second instruction loads the superclass where the method is\nresolved.

    \n
    3. \n
  3. \n

    Then the new OP_GET_SUPER instuction encodes the name of the method to\naccess as an operand.

    \n
    4. \n
\n

The remaining instructions are the normal bytecode for evaluating an argument\nlist and calling a function.

\n

We’re almost ready to implement the new OP_GET_SUPER instruction in the\ninterpreter. But before we do, the compiler has some errors it is responsible\nfor reporting.

\n
static void super_(bool canAssign) {\n
compiler.c
\nin super_()
\n
  if (currentClass == NULL) {\n    error("Can't use 'super' outside of a class.");\n  } else if (!currentClass->hasSuperclass) {\n    error("Can't use 'super' in a class with no superclass.");\n  }\n\n
  consume(TOKEN_DOT, "Expect '.' after 'super'.");\n
\n
compiler.c, in super_()
\n\n

A super call is meaningful only inside the body of a method (or in a function\nnested inside a method), and only inside the method of a class that has a\nsuperclass. We detect both of these cases using the value of currentClass. If\nthat’s NULL or points to a class with no superclass, we report those errors.

\n

29 . 3 . 1Executing super accesses

\n

Assuming the user didn’t put a super expression where it’s not allowed, their\ncode passes from the compiler over to the runtime. We’ve got ourselves a new\ninstruction.

\n
  OP_SET_PROPERTY,\n
chunk.h
\nin enum OpCode
\n
  OP_GET_SUPER,\n
  OP_EQUAL,\n
\n
chunk.h, in enum OpCode
\n\n

We disassemble it like other opcodes that take a constant table index operand.

\n
      return constantInstruction("OP_SET_PROPERTY", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_GET_SUPER:\n      return constantInstruction("OP_GET_SUPER", chunk, offset);\n
    case OP_EQUAL:\n
\n
debug.c, in disassembleInstruction()
\n\n

You might anticipate something harder, but interpreting the new instruction is\nsimilar to executing a normal property access.

\n
      }\n
vm.c
\nin run()
\n
      case OP_GET_SUPER: {\n        ObjString* name = READ_STRING();\n        ObjClass* superclass = AS_CLASS(pop());\n\n        if (!bindMethod(superclass, name)) {\n          return INTERPRET_RUNTIME_ERROR;\n        }\n        break;\n      }\n
      case OP_EQUAL: {\n
\n
vm.c, in run()
\n\n

As with properties, we read the method name from the\nconstant table. Then we pass that to bindMethod() which looks up the method in\nthe given class’s method table and creates an ObjBoundMethod to bundle the\nresulting closure to the current instance.

\n

The key difference is which class we pass to\nbindMethod(). With a normal property access, we use the ObjInstances’s own\nclass, which gives us the dynamic dispatch we want. For a super call, we don’t\nuse the instance’s class. Instead, we use the statically resolved superclass of\nthe containing class, which the compiler has conveniently ensured is sitting on\ntop of the stack waiting for us.

\n

We pop that superclass and pass it to bindMethod(), which correctly skips over\nany overriding methods in any of the subclasses between that superclass and the\ninstance’s own class. It also correctly includes any methods inherited by the\nsuperclass from any of its superclasses.

\n

The rest of the behavior is the same. Popping the superclass leaves the instance\nat the top of the stack. When bindMethod() succeeds, it pops the instance and\npushes the new bound method. Otherwise, it reports a runtime error and returns\nfalse. In that case, we abort the interpreter.

\n\n

29 . 3 . 2Faster super calls

\n

We have superclass method accesses working now. And since the returned object is\nan ObjBoundMethod that you can then invoke, we’ve got super calls working too.\nJust like last chapter, we’ve reached a point where our VM has the complete,\ncorrect semantics.

\n

But, also like last chapter, it’s pretty slow. Again, we’re heap allocating an\nObjBoundMethod for each super call even though most of the time the very next\ninstruction is an OP_CALL that immediately unpacks that bound method, invokes\nit, and then discards it. In fact, this is even more likely to be true for\nsuper calls than for regular method calls. At least with method calls there is\na chance that the user is actually invoking a function stored in a field. With\nsuper calls, you’re always looking up a method. The only question is whether\nyou invoke it immediately or not.

\n

The compiler can certainly answer that question for itself if it sees a left\nparenthesis after the superclass method name, so we’ll go ahead and perform the\nsame optimization we did for method calls. Take out the two lines of code that\nload the superclass and emit OP_GET_SUPER, and replace them with this:

\n
  namedVariable(syntheticToken("this"), false);\n
compiler.c
\nin super_()
\nreplace 2 lines
\n
  if (match(TOKEN_LEFT_PAREN)) {\n    uint8_t argCount = argumentList();\n    namedVariable(syntheticToken("super"), false);\n    emitBytes(OP_SUPER_INVOKE, name);\n    emitByte(argCount);\n  } else {\n    namedVariable(syntheticToken("super"), false);\n    emitBytes(OP_GET_SUPER, name);\n  }\n
}\n
\n
compiler.c, in super_(), replace 2 lines
\n\n

Now before we emit anything, we look for a parenthesized argument list. If we\nfind one, we compile that. Then we load the superclass. After that, we emit a\nnew OP_SUPER_INVOKE instruction. This superinstruction combines the behavior of\nOP_GET_SUPER and OP_CALL, so it takes two operands: the constant table index\nof the method name to look up and the number of arguments to pass to it.

\n\n

Otherwise, if we don’t find a (, we continue to compile the expression as a\nsuper access like we did before and emit an OP_GET_SUPER.

\n

Drifting down the compilation pipeline, our first stop is a new instruction.

\n
  OP_INVOKE,\n
chunk.h
\nin enum OpCode
\n
  OP_SUPER_INVOKE,\n
  OP_CLOSURE,\n
\n
chunk.h, in enum OpCode
\n\n

And just past that, its disassembler support.

\n
      return invokeInstruction("OP_INVOKE", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_SUPER_INVOKE:\n      return invokeInstruction("OP_SUPER_INVOKE", chunk, offset);\n
    case OP_CLOSURE: {\n
\n
debug.c, in disassembleInstruction()
\n\n

A super invocation instruction has the same set of operands as OP_INVOKE, so\nwe reuse the same helper to disassemble it. Finally, the pipeline dumps us into\nthe interpreter.

\n
        break;\n      }\n
vm.c
\nin run()
\n
      case OP_SUPER_INVOKE: {\n        ObjString* method = READ_STRING();\n        int argCount = READ_BYTE();\n        ObjClass* superclass = AS_CLASS(pop());\n        if (!invokeFromClass(superclass, method, argCount)) {\n          return INTERPRET_RUNTIME_ERROR;\n        }\n        frame = &vm.frames[vm.frameCount - 1];\n        break;\n      }\n
      case OP_CLOSURE: {\n
\n
vm.c, in run()
\n\n

This handful of code is basically our implementation of OP_INVOKE mixed\ntogether with a dash of OP_GET_SUPER. There are some differences in how the\nstack is organized, though. With an unoptimized super call, the superclass is\npopped and replaced by the ObjBoundMethod for the resolved function before the\narguments to the call are executed. This ensures that by the time the OP_CALL\nis executed, the bound method is under the argument list, where the runtime\nexpects it to be for a closure call.

\n

With our optimized instructions, things are shuffled a bit:

\"The\n

Now resolving the superclass method is part of the invocation, so the\narguments need to already be on the stack at the point that we look up the\nmethod. This means the superclass object is on top of the arguments.

\n

Aside from that, the behavior is roughly the same as an OP_GET_SUPER followed\nby an OP_CALL. First, we pull out the method name and argument count operands.\nThen we pop the superclass off the top of the stack so that we can look up the\nmethod in its method table. This conveniently leaves the stack set up just right\nfor a method call.

\n

We pass the superclass, method name, and argument count to our existing\ninvokeFromClass() function. That function looks up the given method on the\ngiven class and attempts to create a call to it with the given arity. If a\nmethod could not be found, it returns false, and we bail out of the\ninterpreter. Otherwise, invokeFromClass() pushes a new CallFrame onto the call\nstack for the method’s closure. That invalidates the interpreter’s cached\nCallFrame pointer, so we refresh frame.

\n

29 . 4A Complete Virtual Machine

\n

Take a look back at what we’ve created. By my count, we wrote around 2,500 lines\nof fairly clean, straightforward C. That little program contains a complete\nimplementation of the—quite high-level!—Lox language, with a whole\nprecedence table full of expression types and a suite of control flow\nstatements. We implemented variables, functions, closures, classes, fields,\nmethods, and inheritance.

\n

Even more impressive, our implementation is portable to any platform with a C\ncompiler, and is fast enough for real-world production use. We have a\nsingle-pass bytecode compiler, a tight virtual machine interpreter for our\ninternal instruction set, compact object representations, a stack for storing\nvariables without heap allocation, and a precise garbage collector.

\n

If you go out and start poking around in the implementations of Lua, Python, or\nRuby, you will be surprised by how much of it now looks familiar to you. You\nhave seriously leveled up your knowledge of how programming languages work,\nwhich in turn gives you a deeper understanding of programming itself. It’s like\nyou used to be a race car driver, and now you can pop the hood and repair the\nengine too.

\n

You can stop here if you like. The two implementations of Lox you have are\ncomplete and full featured. You built the car and can drive it wherever you want\nnow. But if you are looking to have more fun tuning and tweaking for even\ngreater performance out on the track, there is one more chapter. We don’t add\nany new capabilities, but we roll in a couple of classic optimizations to\nsqueeze even more perf out. If that sounds fun, keep reading . . . 

\n
\n

Challenges

\n
    \n
  1. \n

    A tenet of object-oriented programming is that a class should ensure new\nobjects are in a valid state. In Lox, that means defining an initializer\nthat populates the instance’s fields. Inheritance complicates invariants\nbecause the instance must be in a valid state according to all of the\nclasses in the object’s inheritance chain.

    \n

    The easy part is remembering to call super.init() in each subclass’s\ninit() method. The harder part is fields. There is nothing preventing two\nclasses in the inheritance chain from accidentally claiming the same field\nname. When this happens, they will step on each other’s fields and possibly\nleave you with an instance in a broken state.

    \n

    If Lox was your language, how would you address this, if at all? If you\nwould change the language, implement your change.

    \n
    2. \n
  2. \n

    Our copy-down inheritance optimization is valid only because Lox does not\npermit you to modify a class’s methods after its declaration. This means we\ndon’t have to worry about the copied methods in the subclass getting out of\nsync with later changes to the superclass.

    \n

    Other languages, like Ruby, do allow classes to be modified after the\nfact. How do implementations of languages like that support class\nmodification while keeping method resolution efficient?

    \n
    3. \n
  3. \n

    In the jlox chapter on inheritance, we had a challenge to\nimplement the BETA language’s approach to method overriding. Solve the\nchallenge again, but this time in clox. Here’s the description of the\nprevious challenge:

    \n

    In Lox, as in most other object-oriented languages, when looking up a\nmethod, we start at the bottom of the class hierarchy and work our way up—a subclass’s method is preferred over a superclass’s. In order to get to the\nsuperclass method from within an overriding method, you use super.

    \n

    The language BETA takes the opposite approach. When you call a\nmethod, it starts at the top of the class hierarchy and works down. A\nsuperclass method wins over a subclass method. In order to get to the\nsubclass method, the superclass method can call inner, which is sort of\nlike the inverse of super. It chains to the next method down the\nhierarchy.

    \n

    The superclass method controls when and where the subclass is allowed to\nrefine its behavior. If the superclass method doesn’t call inner at all,\nthen the subclass has no way of overriding or modifying the superclass’s\nbehavior.

    \n

    Take out Lox’s current overriding and super behavior, and replace it with\nBETA’s semantics. In short:

    \n
      \n
    • \n

      When calling a method on a class, the method highest on the\nclass’s inheritance chain takes precedence.

      \n
      • \n
    • \n

      Inside the body of a method, a call to inner looks for a method with\nthe same name in the nearest subclass along the inheritance chain\nbetween the class containing the inner and the class of this. If\nthere is no matching method, the inner call does nothing.

      \n
      • \n
    \n

    For example:

    \n
    class Doughnut {\n  cook() {\n    print "Fry until golden brown.";\n    inner();\n    print "Place in a nice box.";\n  }\n}\n\nclass BostonCream < Doughnut {\n  cook() {\n    print "Pipe full of custard and coat with chocolate.";\n  }\n}\n\nBostonCream().cook();\n
    \n

    This should print:

    \n
    Fry until golden brown.\nPipe full of custard and coat with chocolate.\nPlace in a nice box.\n
    \n

    Since clox is about not just implementing Lox, but doing so with good\nperformance, this time around try to solve the challenge with an eye towards\nefficiency.

    \n
    4. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/the-lox-language.html", "content": "\n\n\n\nThe Lox Language · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
3
\n

The Lox Language

\n\n
\n

What nicer thing can you do for somebody than make them breakfast?

\n

Anthony Bourdain

\n
\n

We’ll spend the rest of this book illuminating every dark and sundry corner of\nthe Lox language, but it seems cruel to have you immediately start grinding out\ncode for the interpreter without at least a glimpse of what we’re going to end\nup with.

\n

At the same time, I don’t want to drag you through reams of language lawyering\nand specification-ese before you get to touch your text editor. So this will be a gentle, friendly introduction to\nLox. It will leave out a lot of details and edge cases. We’ve got plenty of time\nfor those later.

\n\n

3 . 1Hello, Lox

\n

Here’s your very first taste of Lox:

\n\n
// Your first Lox program!\nprint "Hello, world!";\n
\n

As that // line comment and the trailing semicolon imply, Lox’s syntax is a\nmember of the C family. (There are no parentheses around the string because\nprint is a built-in statement, and not a library function.)

\n

Now, I won’t claim that C has a great syntax. If we\nwanted something elegant, we’d probably mimic Pascal or Smalltalk. If we wanted\nto go full Scandinavian-furniture-minimalism, we’d do a Scheme. Those all have\ntheir virtues.

\n\n

What C-like syntax has instead is something you’ll often find more valuable\nin a language: familiarity. I know you are already comfortable with that style\nbecause the two languages we’ll be using to implement Lox—Java and C—also inherit it. Using a similar syntax for Lox gives you one less thing to\nlearn.

\n

3 . 2A High-Level Language

\n

While this book ended up bigger than I was hoping, it’s still not big enough to\nfit a huge language like Java in it. In order to fit two complete\nimplementations of Lox in these pages, Lox itself has to be pretty compact.

\n

When I think of languages that are small but useful, what comes to mind are\nhigh-level “scripting” languages like JavaScript, Scheme,\nand Lua. Of those three, Lox looks most like JavaScript, mainly because most\nC-syntax languages do. As we’ll learn later, Lox’s approach to scoping hews\nclosely to Scheme. The C flavor of Lox we’ll build in Part III is heavily\nindebted to Lua’s clean, efficient implementation.

\n\n

Lox shares two other aspects with those three languages:

\n

3 . 2 . 1Dynamic typing

\n

Lox is dynamically typed. Variables can store values of any type, and a single\nvariable can even store values of different types at different times. If you try\nto perform an operation on values of the wrong type—say, dividing a number by\na string—then the error is detected and reported at runtime.

\n

There are plenty of reasons to like static types, but\nthey don’t outweigh the pragmatic reasons to pick dynamic types for Lox. A\nstatic type system is a ton of work to learn and implement. Skipping it gives\nyou a simpler language and a shorter book. We’ll get our interpreter up and\nexecuting bits of code sooner if we defer our type checking to runtime.

\n\n

3 . 2 . 2Automatic memory management

\n

High-level languages exist to eliminate error-prone, low-level drudgery, and what\ncould be more tedious than manually managing the allocation and freeing of\nstorage? No one rises and greets the morning sun with, “I can’t wait to figure\nout the correct place to call free() for every byte of memory I allocate\ntoday!”

\n

There are two main techniques for managing memory:\nreference counting and tracing garbage collection (usually just called\ngarbage collection or GC). Ref counters are much simpler to implement—I think that’s why Perl, PHP, and Python all started out using them. But, over\ntime, the limitations of ref counting become too troublesome. All of those\nlanguages eventually ended up adding a full tracing GC, or at least enough of\none to clean up object cycles.

\n\n

Tracing garbage collection has a fearsome reputation. It is a little harrowing\nworking at the level of raw memory. Debugging a GC can sometimes leave you\nseeing hex dumps in your dreams. But, remember, this book is about dispelling\nmagic and slaying those monsters, so we are going to write our own garbage\ncollector. I think you’ll find the algorithm is quite simple and a lot of fun to\nimplement.

\n

3 . 3Data Types

\n

In Lox’s little universe, the atoms that make up all matter are the built-in\ndata types. There are only a few:

\n
    \n
  • \n

    Booleans. You can’t code without logic and you\ncan’t logic without Boolean values. “True” and “false”, the yin and yang of\nsoftware. Unlike some ancient languages that repurpose an existing type to\nrepresent truth and falsehood, Lox has a dedicated Boolean type. We may\nbe roughing it on this expedition, but we aren’t savages.

    \n\n

    There are two Boolean values, obviously, and a literal for each one.

    \n
    true;  // Not false.\nfalse; // Not *not* false.\n
    \n
    • \n
  • \n

    Numbers. Lox has only one kind of number: double-precision floating\npoint. Since floating-point numbers can also represent a wide range of\nintegers, that covers a lot of territory, while keeping things simple.

    \n

    Full-featured languages have lots of syntax for numbers—hexadecimal,\nscientific notation, octal, all sorts of fun stuff. We’ll settle for basic\ninteger and decimal literals.

    \n
    1234;  // An integer.\n12.34; // A decimal number.\n
    \n
    • \n
  • \n

    Strings. We’ve already seen one string literal in the first example.\nLike most languages, they are enclosed in double quotes.

    \n
    "I am a string";\n"";    // The empty string.\n"123"; // This is a string, not a number.\n
    \n

    As we’ll see when we get to implementing them, there is quite a lot of\ncomplexity hiding in that innocuous sequence of characters.

    \n
    • \n
  • \n

    Nil. There’s one last built-in value who’s never invited to the party\nbut always seems to show up. It represents “no value”. It’s called “null” in\nmany other languages. In Lox we spell it nil. (When we get to implementing\nit, that will help distinguish when we’re talking about Lox’s nil versus\nJava or C’s null.)

    \n

    There are good arguments for not having a null value in a language since\nnull pointer errors are the scourge of our industry. If we were doing a\nstatically typed language, it would be worth trying to ban it. In a\ndynamically typed one, though, eliminating it is often more annoying\nthan having it.

    \n
    • \n
\n

3 . 4Expressions

\n

If built-in data types and their literals are atoms, then expressions must\nbe the molecules. Most of these will be familiar.

\n

3 . 4 . 1Arithmetic

\n

Lox features the basic arithmetic operators you know and love from C and other\nlanguages:

\n
add + me;\nsubtract - me;\nmultiply * me;\ndivide / me;\n
\n

The subexpressions on either side of the operator are operands. Because\nthere are two of them, these are called binary operators. (It has nothing\nto do with the ones-and-zeroes use of “binary”.) Because the operator is fixed in the middle of the operands, these are also\ncalled infix operators (as opposed to prefix operators where the\noperator comes before the operands, and postfix where it comes after).

\n\n

One arithmetic operator is actually both an infix and a prefix one. The -\noperator can also be used to negate a number.

\n
-negateMe;\n
\n

All of these operators work on numbers, and it’s an error to pass any other\ntypes to them. The exception is the + operator—you can also pass it two\nstrings to concatenate them.

\n

3 . 4 . 2Comparison and equality

\n

Moving along, we have a few more operators that always return a Boolean result.\nWe can compare numbers (and only numbers), using Ye Olde Comparison Operators.

\n
less < than;\nlessThan <= orEqual;\ngreater > than;\ngreaterThan >= orEqual;\n
\n

We can test two values of any kind for equality or inequality.

\n
1 == 2;         // false.\n"cat" != "dog"; // true.\n
\n

Even different types.

\n
314 == "pi"; // false.\n
\n

Values of different types are never equivalent.

\n
123 == "123"; // false.\n
\n

I’m generally against implicit conversions.

\n

3 . 4 . 3Logical operators

\n

The not operator, a prefix !, returns false if its operand is true, and vice\nversa.

\n
!true;  // false.\n!false; // true.\n
\n

The other two logical operators really are control flow constructs in the guise\nof expressions. An and expression determines if two\nvalues are both true. It returns the left operand if it’s false, or the\nright operand otherwise.

\n
true and false; // false.\ntrue and true;  // true.\n
\n

And an or expression determines if either of two values (or both) are true.\nIt returns the left operand if it is true and the right operand otherwise.

\n
false or false; // false.\ntrue or false;  // true.\n
\n\n

The reason and and or are like control flow structures is that they\nshort-circuit. Not only does and return the left operand if it is false,\nit doesn’t even evaluate the right one in that case. Conversely\n(contrapositively?), if the left operand of an or is true, the right is\nskipped.

\n

3 . 4 . 4Precedence and grouping

\n

All of these operators have the same precedence and associativity that you’d\nexpect coming from C. (When we get to parsing, we’ll get way more precise\nabout that.) In cases where the precedence isn’t what you want, you can use ()\nto group stuff.

\n
var average = (min + max) / 2;\n
\n

Since they aren’t very technically interesting, I’ve cut the remainder of the\ntypical operator menagerie out of our little language. No bitwise, shift,\nmodulo, or conditional operators. I’m not grading you, but you will get bonus\npoints in my heart if you augment your own implementation of Lox with them.

\n

Those are the expression forms (except for a couple related to specific features\nthat we’ll get to later), so let’s move up a level.

\n

3 . 5Statements

\n

Now we’re at statements. Where an expression’s main job is to produce a value,\na statement’s job is to produce an effect. Since, by definition, statements\ndon’t evaluate to a value, to be useful they have to otherwise change the world\nin some way—usually modifying some state, reading input, or producing output.

\n

You’ve seen a couple of kinds of statements already. The first one was:

\n
print "Hello, world!";\n
\n

A print statement evaluates a single expression\nand displays the result to the user. You’ve also seen some statements like:

\n\n
"some expression";\n
\n

An expression followed by a semicolon (;) promotes the expression to\nstatement-hood. This is called (imaginatively enough), an expression\nstatement.

\n

If you want to pack a series of statements where a single one is expected, you\ncan wrap them up in a block.

\n
{\n  print "One statement.";\n  print "Two statements.";\n}\n
\n

Blocks also affect scoping, which leads us to the next section . . . 

\n

3 . 6Variables

\n

You declare variables using var statements. If you omit the initializer, the variable’s value defaults to nil.

\n\n
var imAVariable = "here is my value";\nvar iAmNil;\n
\n

Once declared, you can, naturally, access and assign a variable using its name.

\n

\n
var breakfast = "bagels";\nprint breakfast; // "bagels".\nbreakfast = "beignets";\nprint breakfast; // "beignets".\n
\n\n

I won’t get into the rules for variable scope here, because we’re going to spend\na surprising amount of time in later chapters mapping every square inch of the\nrules. In most cases, it works like you would expect coming from C or Java.

\n

3 . 7Control Flow

\n

It’s hard to write useful programs if you can’t skip\nsome code or execute some more than once. That means control flow. In addition\nto the logical operators we already covered, Lox lifts three statements straight\nfrom C.

\n\n

An if statement executes one of two statements based on some condition.

\n
if (condition) {\n  print "yes";\n} else {\n  print "no";\n}\n
\n

A while loop executes the body repeatedly as long as\nthe condition expression evaluates to true.

\n
var a = 1;\nwhile (a < 10) {\n  print a;\n  a = a + 1;\n}\n
\n\n

Finally, we have for loops.

\n
for (var a = 1; a < 10; a = a + 1) {\n  print a;\n}\n
\n

This loop does the same thing as the previous while loop. Most modern\nlanguages also have some sort of for-in or\nforeach loop for explicitly iterating over various sequence types. In a real\nlanguage, that’s nicer than the crude C-style for loop we got here. Lox keeps\nit basic.

\n\n

3 . 8Functions

\n

A function call expression looks the same as it does in C.

\n
makeBreakfast(bacon, eggs, toast);\n
\n

You can also call a function without passing anything to it.

\n
makeBreakfast();\n
\n

Unlike in, say, Ruby, the parentheses are mandatory in this case. If you leave them\noff, the name doesn’t call the function, it just refers to it.

\n

A language isn’t very fun if you can’t define your own functions. In Lox, you do\nthat with fun.

\n\n
fun printSum(a, b) {\n  print a + b;\n}\n
\n

Now’s a good time to clarify some terminology. Some\npeople throw around “parameter” and “argument” like they are interchangeable\nand, to many, they are. We’re going to spend a lot of time splitting the finest\nof downy hairs around semantics, so let’s sharpen our words. From here on out:

\n
    \n
  • \n

    An argument is an actual value you pass to a function when you call it.\nSo a function call has an argument list. Sometimes you hear actual\nparameter used for these.

    \n
    • \n
  • \n

    A parameter is a variable that holds the value of the argument inside\nthe body of the function. Thus, a function declaration has a parameter\nlist. Others call these formal parameters or simply formals.

    \n
    • \n
\n\n

The body of a function is always a block. Inside it, you can return a value\nusing a return statement.

\n
fun returnSum(a, b) {\n  return a + b;\n}\n
\n

If execution reaches the end of the block without hitting a return, it\nimplicitly returns nil.

\n\n

3 . 8 . 1Closures

\n

Functions are first class in Lox, which just means they are real values that\nyou can get a reference to, store in variables, pass around, etc. This works:

\n
fun addPair(a, b) {\n  return a + b;\n}\n\nfun identity(a) {\n  return a;\n}\n\nprint identity(addPair)(1, 2); // Prints "3".\n
\n

Since function declarations are statements, you can declare local functions\ninside another function.

\n
fun outerFunction() {\n  fun localFunction() {\n    print "I'm local!";\n  }\n\n  localFunction();\n}\n
\n

If you combine local functions, first-class functions, and block scope, you run\ninto this interesting situation:

\n
fun returnFunction() {\n  var outside = "outside";\n\n  fun inner() {\n    print outside;\n  }\n\n  return inner;\n}\n\nvar fn = returnFunction();\nfn();\n
\n

Here, inner() accesses a local variable declared outside of its body in the\nsurrounding function. Is this kosher? Now that lots of languages have borrowed\nthis feature from Lisp, you probably know the answer is yes.

\n

For that to work, inner() has to “hold on” to references to any surrounding\nvariables that it uses so that they stay around even after the outer function\nhas returned. We call functions that do this closures. These days, the term is often used for any\nfirst-class function, though it’s sort of a misnomer if the function doesn’t\nhappen to close over any variables.

\n\n

As you can imagine, implementing these adds some complexity because we can no\nlonger assume variable scope works strictly like a stack where local variables\nevaporate the moment the function returns. We’re going to have a fun time\nlearning how to make these work correctly and efficiently.

\n

3 . 9Classes

\n

Since Lox has dynamic typing, lexical (roughly, “block”) scope, and closures,\nit’s about halfway to being a functional language. But as you’ll see, it’s\nalso about halfway to being an object-oriented language. Both paradigms have a\nlot going for them, so I thought it was worth covering some of each.

\n

Since classes have come under fire for not living up to their hype, let me first\nexplain why I put them into Lox and this book. There are really two questions:

\n

3 . 9 . 1Why might any language want to be object oriented?

\n

Now that object-oriented languages like Java have sold out and only play arena\nshows, it’s not cool to like them anymore. Why would anyone make a new\nlanguage with objects? Isn’t that like releasing music on 8-track?

\n

It is true that the “all inheritance all the time” binge of the ’90s produced\nsome monstrous class hierarchies, but object-oriented programming (OOP)\nis still pretty rad. Billions of lines of successful code have been written in\nOOP languages, shipping millions of apps to happy users. Likely a majority of\nworking programmers today are using an object-oriented language. They can’t all\nbe that wrong.

\n

In particular, for a dynamically typed language, objects are pretty handy. We\nneed some way of defining compound data types to bundle blobs of stuff\ntogether.

\n

If we can also hang methods off of those, then we avoid the need to prefix all\nof our functions with the name of the data type they operate on to avoid\ncolliding with similar functions for different types. In, say, Racket, you end\nup having to name your functions like hash-copy (to copy a hash table) and\nvector-copy (to copy a vector) so that they don’t step on each other. Methods\nare scoped to the object, so that problem goes away.

\n

3 . 9 . 2Why is Lox object oriented?

\n

I could claim objects are groovy but still out of scope for the book. Most\nprogramming language books, especially ones that try to implement a whole\nlanguage, leave objects out. To me, that means the topic isn’t well covered.\nWith such a widespread paradigm, that omission makes me sad.

\n

Given how many of us spend all day using OOP languages, it seems like the\nworld could use a little documentation on how to make one. As you’ll see, it\nturns out to be pretty interesting. Not as hard as you might fear, but not as\nsimple as you might presume, either.

\n

3 . 9 . 3Classes or prototypes

\n

When it comes to objects, there are actually two approaches to them, classes\nand prototypes. Classes came first, and are more common thanks to C++, Java,\nC#, and friends. Prototypes were a virtually forgotten offshoot until JavaScript\naccidentally took over the world.

\n

In class-based languages, there are two core concepts: instances and classes.\nInstances store the state for each object and have a reference to the instance’s\nclass. Classes contain the methods and inheritance chain. To call a method on an\ninstance, there is always a level of indirection. You look up the instance’s class and then you find the method\nthere:

\n\"How\n

Prototype-based languages merge these two concepts.\nThere are only objects—no classes—and each individual object may contain\nstate and methods. Objects can directly inherit from each other (or “delegate\nto” in prototypal lingo):

\n\"How\n

This means that in some ways prototypal languages are more fundamental than\nclasses. They are really neat to implement because they’re so simple. Also,\nthey can express lots of unusual patterns that classes steer you away from.

\n

But I’ve looked at a lot of code written in prototypal languages—including\nsome of my own devising. Do you know what people generally do with all\nof the power and flexibility of prototypes?  . . . They use them to reinvent\nclasses.

\n

I don’t know why that is, but people naturally seem to prefer a class-based\n(Classic? Classy?) style. Prototypes are simpler in the language, but they\nseem to accomplish that only by pushing the\ncomplexity onto the user. So, for Lox, we’ll save our users the trouble and bake\nclasses right in.

\n\n

3 . 9 . 4Classes in Lox

\n

Enough rationale, let’s see what we actually have. Classes encompass a\nconstellation of features in most languages. For Lox, I’ve selected what I think\nare the brightest stars. You declare a class and its methods like so:

\n
class Breakfast {\n  cook() {\n    print "Eggs a-fryin'!";\n  }\n\n  serve(who) {\n    print "Enjoy your breakfast, " + who + ".";\n  }\n}\n
\n

The body of a class contains its methods. They look like function declarations\nbut without the fun keyword. When the class\ndeclaration is executed, Lox creates a class object and stores that in a\nvariable named after the class. Just like functions, classes are first class in\nLox.

\n\n
// Store it in variables.\nvar someVariable = Breakfast;\n\n// Pass it to functions.\nsomeFunction(Breakfast);\n
\n

Next, we need a way to create instances. We could add some sort of new\nkeyword, but to keep things simple, in Lox the class itself is a factory\nfunction for instances. Call a class like a function, and it produces a new\ninstance of itself.

\n
var breakfast = Breakfast();\nprint breakfast; // "Breakfast instance".\n
\n

3 . 9 . 5Instantiation and initialization

\n

Classes that only have behavior aren’t super useful. The idea behind\nobject-oriented programming is encapsulating behavior and state together. To\ndo that, you need fields. Lox, like other dynamically typed languages, lets you\nfreely add properties onto objects.

\n
breakfast.meat = "sausage";\nbreakfast.bread = "sourdough";\n
\n

Assigning to a field creates it if it doesn’t already exist.

\n

If you want to access a field or method on the current object from within a\nmethod, you use good old this.

\n
class Breakfast {\n  serve(who) {\n    print "Enjoy your " + this.meat + " and " +\n        this.bread + ", " + who + ".";\n  }\n\n  // ...\n}\n
\n

Part of encapsulating data within an object is ensuring the object is in a valid\nstate when it’s created. To do that, you can define an initializer. If your\nclass has a method named init(), it is called automatically when the object is\nconstructed. Any parameters passed to the class are forwarded to its\ninitializer.

\n
class Breakfast {\n  init(meat, bread) {\n    this.meat = meat;\n    this.bread = bread;\n  }\n\n  // ...\n}\n\nvar baconAndToast = Breakfast("bacon", "toast");\nbaconAndToast.serve("Dear Reader");\n// "Enjoy your bacon and toast, Dear Reader."\n
\n

3 . 9 . 6Inheritance

\n

Every object-oriented language lets you not only define methods, but reuse them\nacross multiple classes or objects. For that, Lox supports single inheritance.\nWhen you declare a class, you can specify a class that it inherits from using a less-than\n(<) operator.

\n
class Brunch < Breakfast {\n  drink() {\n    print "How about a Bloody Mary?";\n  }\n}\n
\n\n

Here, Brunch is the derived class or subclass, and Breakfast is the\nbase class or superclass.

\n

Every method defined in the superclass is also available to its subclasses.

\n
var benedict = Brunch("ham", "English muffin");\nbenedict.serve("Noble Reader");\n
\n

Even the init() method gets inherited. In practice,\nthe subclass usually wants to define its own init() method too. But the\noriginal one also needs to be called so that the superclass can maintain its\nstate. We need some way to call a method on our own instance without hitting\nour own methods.

\n\n

As in Java, you use super for that.

\n
class Brunch < Breakfast {\n  init(meat, bread, drink) {\n    super.init(meat, bread);\n    this.drink = drink;\n  }\n}\n
\n

That’s about it for object orientation. I tried to keep the feature set minimal.\nThe structure of the book did force one compromise. Lox is not a pure\nobject-oriented language. In a true OOP language every object is an instance of\na class, even primitive values like numbers and Booleans.

\n

Because we don’t implement classes until well after we start working with the\nbuilt-in types, that would have been hard. So values of primitive types aren’t\nreal objects in the sense of being instances of classes. They don’t have methods\nor properties. If I were trying to make Lox a real language for real users, I\nwould fix that.

\n

3 . 10The Standard Library

\n

We’re almost done. That’s the whole language, so all that’s left is the “core”\nor “standard” library—the set of functionality that is implemented directly\nin the interpreter and that all user-defined behavior is built on top of.

\n

This is the saddest part of Lox. Its standard library goes beyond minimalism and\nveers close to outright nihilism. For the sample code in the book, we only need\nto demonstrate that code is running and doing what it’s supposed to do. For\nthat, we already have the built-in print statement.

\n

Later, when we start optimizing, we’ll write some benchmarks and see how long it\ntakes to execute code. That means we need to track time, so we’ll define one\nbuilt-in function, clock(), that returns the number of seconds since the\nprogram started.

\n

And . . . that’s it. I know, right? It’s embarrassing.

\n

If you wanted to turn Lox into an actual useful language, the very first thing\nyou should do is flesh this out. String manipulation, trigonometric functions,\nfile I/O, networking, heck, even reading input from the user would help. But we\ndon’t need any of that for this book, and adding it wouldn’t teach you anything\ninteresting, so I’ve left it out.

\n

Don’t worry, we’ll have plenty of exciting stuff in the language itself to keep\nus busy.

\n
\n

Challenges

\n
    \n
  1. \n

    Write some sample Lox programs and run them (you can use the implementations\nof Lox in my repository). Try to come up with edge case behavior I\ndidn’t specify here. Does it do what you expect? Why or why not?

    \n
    2. \n
  2. \n

    This informal introduction leaves a lot unspecified. List several open\nquestions you have about the language’s syntax and semantics. What do you\nthink the answers should be?

    \n
    3. \n
  3. \n

    Lox is a pretty tiny language. What features do you think it is missing that\nwould make it annoying to use for real programs? (Aside from the standard\nlibrary, of course.)

    \n
    4. \n
\n
\n
\n

Design Note: Expressions and Statements

\n

Lox has both expressions and statements. Some languages omit the latter.\nInstead, they treat declarations and control flow constructs as expressions too.\nThese “everything is an expression” languages tend to have functional pedigrees\nand include most Lisps, SML, Haskell, Ruby, and CoffeeScript.

\n

To do that, for each “statement-like” construct in the language, you need to\ndecide what value it evaluates to. Some of those are easy:

\n
    \n
  • \n

    An if expression evaluates to the result of whichever branch is chosen.\nLikewise, a switch or other multi-way branch evaluates to whichever case\nis picked.

    \n
    • \n
  • \n

    A variable declaration evaluates to the value of the variable.

    \n
    • \n
  • \n

    A block evaluates to the result of the last expression in the sequence.

    \n
    • \n
\n

Some get a little stranger. What should a loop evaluate to? A while loop in\nCoffeeScript evaluates to an array containing each element that the body\nevaluated to. That can be handy, or a waste of memory if you don’t need the\narray.

\n

You also have to decide how these statement-like expressions compose with other\nexpressions—you have to fit them into the grammar’s precedence table. For\nexample, Ruby allows:

\n
puts 1 + if true then 2 else 3 end + 4\n
\n

Is this what you’d expect? Is it what your users expect? How does this affect\nhow you design the syntax for your “statements”? Note that Ruby has an explicit\nend to tell when the if expression is complete. Without it, the + 4 would\nlikely be parsed as part of the else clause.

\n

Turning every statement into an expression forces you to answer a few hairy\nquestions like that. In return, you eliminate some redundancy. C has both blocks\nfor sequencing statements, and the comma operator for sequencing expressions. It\nhas both the if statement and the ?: conditional operator. If everything was\nan expression in C, you could unify each of those.

\n

Languages that do away with statements usually also feature implicit returns—a function automatically returns whatever value its body evaluates to without\nneed for some explicit return syntax. For small functions and methods, this is\nreally handy. In fact, many languages that do have statements have added syntax\nlike => to be able to define functions whose body is the result of evaluating\na single expression.

\n

But making all functions work that way can be a little strange. If you aren’t\ncareful, your function will leak a return value even if you only intend it to\nproduce a side effect. In practice, though, users of these languages don’t find\nit to be a problem.

\n

For Lox, I gave it statements for prosaic reasons. I picked a C-like syntax for\nfamiliarity’s sake, and trying to take the existing C statement syntax and\ninterpret it like expressions gets weird pretty fast.

\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/types-of-values.html", "content": "\n\n\n\nTypes of Values · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
18
\n

Types of Values

\n\n
\n

When you are a Bear of Very Little Brain, and you Think of Things, you find\nsometimes that a Thing which seemed very Thingish inside you is quite\ndifferent when it gets out into the open and has other people looking at it.

\n

A. A. Milne, Winnie-the-Pooh

\n
\n

The past few chapters were huge, packed full of complex techniques and pages of\ncode. In this chapter, there’s only one new concept to learn and a scattering of\nstraightforward code. You’ve earned a respite.

\n

Lox is dynamically typed. A single variable can\nhold a Boolean, number, or string at different points in time. At least, that’s\nthe idea. Right now, in clox, all values are numbers. By the end of the chapter,\nit will also support Booleans and nil. While those aren’t super interesting,\nthey force us to figure out how our value representation can dynamically handle\ndifferent types.

\n\n

18 . 1Tagged Unions

\n

The nice thing about working in C is that we can build our data structures from\nthe raw bits up. The bad thing is that we have to do that. C doesn’t give you\nmuch for free at compile time and even less at runtime. As far as C is\nconcerned, the universe is an undifferentiated array of bytes. It’s up to us to\ndecide how many of those bytes to use and what they mean.

\n

In order to choose a value representation, we need to answer two key questions:

\n
    \n
  1. \n

    How do we represent the type of a value? If you try to, say, multiply a\nnumber by true, we need to detect that error at runtime and report it. In\norder to do that, we need to be able to tell what a value’s type is.

    \n
    2. \n
  2. \n

    How do we store the value itself? We need to not only be able to tell\nthat three is a number, but that it’s different from the number four. I\nknow, seems obvious, right? But we’re operating at a level where it’s good\nto spell these things out.

    \n
    3. \n
\n

Since we’re not just designing this language but building it ourselves, when\nanswering these two questions we also have to keep in mind the implementer’s\neternal quest: to do it efficiently.

\n

Language hackers over the years have come up with a variety of clever ways to\npack the above information into as few bits as possible. For now, we’ll start\nwith the simplest, classic solution: a tagged union. A value contains two\nparts: a type “tag”, and a payload for the actual value. To store the value’s\ntype, we define an enum for each kind of value the VM supports.

\n
#include "common.h"\n\n
value.h
\n
typedef enum {\n  VAL_BOOL,\n  VAL_NIL, \n  VAL_NUMBER,\n} ValueType;\n\n
typedef double Value;\n
\n
value.h
\n\n\n

For now, we have only a couple of cases, but this will grow as we add strings,\nfunctions, and classes to clox. In addition to the type, we also need to store\nthe data for the value—the double for a number, true or false for a\nBoolean. We could define a struct with fields for each possible type.

\"A\n

But this is a waste of memory. A value can’t simultaneously be both a number and\na Boolean. So at any point in time, only one of those fields will be used. C\nlets you optimize this by defining a union. A union\nlooks like a struct except that all of its fields overlap in memory.

\n\"A\n

The size of a union is the size of its largest field. Since the fields all reuse\nthe same bits, you have to be very careful when working with them. If you store\ndata using one field and then access it using another, you will reinterpret what the underlying bits\nmean.

\n\n

As the name “tagged union” implies, our new value representation combines these\ntwo parts into a single struct.

\n
} ValueType;\n\n
value.h
\nadd after enum ValueType
\nreplace 1 line
\n
typedef struct {\n  ValueType type;\n  union {\n    bool boolean;\n    double number;\n  } as; \n} Value;\n
\n\ntypedef struct {\n
\n
value.h, add after enum ValueType, replace 1 line
\n\n

There’s a field for the type tag, and then a second field containing the union\nof all of the underlying values. On a 64-bit machine with a typical C compiler,\nthe layout looks like this:

\n\"The\n

The four-byte type tag comes first, then the union. Most architectures prefer\nvalues be aligned to their size. Since the union field contains an eight-byte\ndouble, the compiler adds four bytes of padding after\nthe type field to keep that double on the nearest eight-byte boundary. That\nmeans we’re effectively spending eight bytes on the type tag, which only needs\nto represent a number between zero and three. We could stuff the enum in a\nsmaller size, but all that would do is increase the padding.

\n\n

So our Values are 16 bytes, which seems a little large. We’ll improve it\nlater. In the meantime, they’re still small enough to store on\nthe C stack and pass around by value. Lox’s semantics allow that because the\nonly types we support so far are immutable. If we pass a copy of a Value\ncontaining the number three to some function, we don’t need to worry about the\ncaller seeing modifications to the value. You can’t “modify” three. It’s three\nforever.

\n

18 . 2Lox Values and C Values

\n

That’s our new value representation, but we aren’t done. Right now, the rest of\nclox assumes Value is an alias for double. We have code that does a straight C\ncast from one to the other. That code is all broken now. So sad.

\n

With our new representation, a Value can contain a double, but it’s not\nequivalent to it. There is a mandatory conversion step to get from one to the\nother. We need to go through the code and insert those conversions to get clox\nworking again.

\n

We’ll implement these conversions as a handful of macros, one for each type and\noperation. First, to promote a native C value to a clox Value:

\n
} Value;\n
value.h
\nadd after struct Value
\n
\n\n#define BOOL_VAL(value)   ((Value){VAL_BOOL, {.boolean = value}})\n#define NIL_VAL           ((Value){VAL_NIL, {.number = 0}})\n#define NUMBER_VAL(value) ((Value){VAL_NUMBER, {.number = value}})\n
\n\ntypedef struct {\n
\n
value.h, add after struct Value
\n\n

Each one of these takes a C value of the appropriate type and produces a Value\nthat has the correct type tag and contains the underlying value. This hoists\nstatically typed values up into clox’s dynamically typed universe. In order to\ndo anything with a Value, though, we need to unpack it and get the C value\nback out.

\n
} Value;\n
value.h
\nadd after struct Value
\n
\n\n#define AS_BOOL(value)    ((value).as.boolean)\n#define AS_NUMBER(value)  ((value).as.number)\n
\n\n#define BOOL_VAL(value)   ((Value){VAL_BOOL, {.boolean = value}})\n
\n
value.h, add after struct Value
\n\n\n

These macros go in the opposite direction. Given a\nValue of the right type, they unwrap it and return the corresponding raw C\nvalue. The “right type” part is important! These macros directly access the\nunion fields. If we were to do something like:

\n
Value value = BOOL_VAL(true);\ndouble number = AS_NUMBER(value);\n
\n

Then we may open a smoldering portal to the Shadow Realm. It’s not safe to use\nany of the AS_ macros unless we know the Value contains the appropriate type.\nTo that end, we define a last few macros to check a Value’s type.

\n
} Value;\n
value.h
\nadd after struct Value
\n
\n\n#define IS_BOOL(value)    ((value).type == VAL_BOOL)\n#define IS_NIL(value)     ((value).type == VAL_NIL)\n#define IS_NUMBER(value)  ((value).type == VAL_NUMBER)\n
\n\n#define AS_BOOL(value)    ((value).as.boolean)\n
\n
value.h, add after struct Value
\n\n

These macros return true if the Value has that\ntype. Any time we call one of the AS_ macros, we need to guard it behind a\ncall to one of these first. With these eight macros, we can now safely shuttle\ndata between Lox’s dynamic world and C’s static one.

\n\n

18 . 3Dynamically Typed Numbers

\n

We’ve got our value representation and the tools to convert to and from it. All\nthat’s left to get clox running again is to grind through the code and fix every\nplace where data moves across that boundary. This is one of those sections of\nthe book that isn’t exactly mind-blowing, but I promised I’d show you every\nsingle line of code, so here we are.

\n

The first values we create are the constants generated when we compile number\nliterals. After we convert the lexeme to a C double, we simply wrap it in a\nValue before storing it in the constant table.

\n
  double value = strtod(parser.previous.start, NULL);\n
compiler.c
\nin number()
\nreplace 1 line
\n
  emitConstant(NUMBER_VAL(value));\n
}\n
\n
compiler.c, in number(), replace 1 line
\n\n

Over in the runtime, we have a function to print values.

\n
void printValue(Value value) {\n
value.c
\nin printValue()
\nreplace 1 line
\n
 printf("%g", AS_NUMBER(value));\n
}\n
\n
value.c, in printValue(), replace 1 line
\n\n

Right before we send the Value to printf(), we unwrap it and extract the\ndouble value. We’ll revisit this function shortly to add the other types, but\nlet’s get our existing code working first.

\n

18 . 3 . 1Unary negation and runtime errors

\n

The next simplest operation is unary negation. It pops a value off the stack,\nnegates it, and pushes the result. Now that we have other types of values, we\ncan’t assume the operand is a number anymore. The user could just as well do:

\n
print -false; // Uh...\n
\n

We need to handle that gracefully, which means it’s time for runtime errors.\nBefore performing an operation that requires a certain type, we need to make\nsure the Value is that type.

\n

For unary negation, the check looks like this:

\n
      case OP_DIVIDE:   BINARY_OP(/); break;\n
vm.c
\nin run()
\nreplace 1 line
\n
      case OP_NEGATE:\n        if (!IS_NUMBER(peek(0))) {\n          runtimeError("Operand must be a number.");\n          return INTERPRET_RUNTIME_ERROR;\n        }\n        push(NUMBER_VAL(-AS_NUMBER(pop())));\n        break;\n
      case OP_RETURN: {\n
\n
vm.c, in run(), replace 1 line
\n\n

First, we check to see if the Value on top of the stack is a number. If it’s\nnot, we report the runtime error and stop the\ninterpreter. Otherwise, we keep going. Only after this validation do we unwrap\nthe operand, negate it, wrap the result and push it.

\n\n

To access the Value, we use a new little function.

\n
vm.c
\nadd after pop()
\n
static Value peek(int distance) {\n  return vm.stackTop[-1 - distance];\n}\n
\n
vm.c, add after pop()
\n\n

It returns a Value from the stack but doesn’t pop it.\nThe distance argument is how far down from the top of the stack to look: zero\nis the top, one is one slot down, etc.

\n\n

We report the runtime error using a new function that we’ll get a lot of mileage\nout of over the remainder of the book.

\n
vm.c
\nadd after resetStack()
\n
static void runtimeError(const char* format, ...) {\n  va_list args;\n  va_start(args, format);\n  vfprintf(stderr, format, args);\n  va_end(args);\n  fputs("\\n", stderr);\n\n  size_t instruction = vm.ip - vm.chunk->code - 1;\n  int line = vm.chunk->lines[instruction];\n  fprintf(stderr, "[line %d] in script\\n", line);\n  resetStack();\n}\n
\n
vm.c, add after resetStack()
\n\n

You’ve certainly called variadic functions—ones that take a varying number\nof arguments—in C before: printf() is one. But you may not have defined\nyour own. This book isn’t a C tutorial, so I’ll\nskim over it here, but basically the ... and va_list stuff let us pass an\narbitrary number of arguments to runtimeError(). It forwards those on to\nvfprintf(), which is the flavor of printf() that takes an explicit\nva_list.

\n\n

Callers can pass a format string to runtimeError() followed by a number of\narguments, just like they can when calling printf() directly. runtimeError()\nthen formats and prints those arguments. We won’t take advantage of that in this\nchapter, but later chapters will produce formatted runtime error messages that\ncontain other data.

\n

After we show the hopefully helpful error message, we tell the user which line of their code was being executed when the error\noccurred. Since we left the tokens behind in the compiler, we look up the line\nin the debug information compiled into the chunk. If our compiler did its job\nright, that corresponds to the line of source code that the bytecode was\ncompiled from.

\n

We look into the chunk’s debug line array using the current bytecode instruction\nindex minus one. That’s because the interpreter advances past each instruction\nbefore executing it. So, at the point that we call runtimeError(), the failed\ninstruction is the previous one.

\n\n

In order to use va_list and the macros for working with it, we need to bring\nin a standard header.

\n
vm.c
\nadd to top of file
\n
#include <stdarg.h>\n
#include <stdio.h>\n
\n
vm.c, add to top of file
\n\n

With this, our VM can not only do the right thing when we negate numbers (like\nit used to before we broke it), but it also gracefully handles erroneous\nattempts to negate other types (which we don’t have yet, but still).

\n

18 . 3 . 2Binary arithmetic operators

\n

We have our runtime error machinery in place now, so fixing the binary operators\nis easier even though they’re more complex. We support four binary operators\ntoday: +, -, *, and /. The only difference between them is which\nunderlying C operator they use. To minimize redundant code between the four\noperators, we wrapped up the commonality in a big preprocessor macro that takes\nthe operator token as a parameter.

\n

That macro seemed like overkill a few chapters ago, but we get the benefit\nfrom it today. It lets us add the necessary type checking and conversions in one\nplace.

\n
#define READ_CONSTANT() (vm.chunk->constants.values[READ_BYTE()])\n
vm.c
\nin run()
\nreplace 6 lines
\n
#define BINARY_OP(valueType, op) \\\n    do { \\\n      if (!IS_NUMBER(peek(0)) || !IS_NUMBER(peek(1))) { \\\n        runtimeError("Operands must be numbers."); \\\n        return INTERPRET_RUNTIME_ERROR; \\\n      } \\\n      double b = AS_NUMBER(pop()); \\\n      double a = AS_NUMBER(pop()); \\\n      push(valueType(a op b)); \\\n    } while (false)\n
\n\n  for (;;) {\n
\n
vm.c, in run(), replace 6 lines
\n\n

Yeah, I realize that’s a monster of a macro. It’s not what I’d normally consider\ngood C practice, but let’s roll with it. The changes are similar to what we did\nfor unary negate. First, we check that the two operands are both numbers. If\neither isn’t, we report a runtime error and yank the ejection seat lever.

\n

If the operands are fine, we pop them both and unwrap them. Then we apply the\ngiven operator, wrap the result, and push it back on the stack. Note that we\ndon’t wrap the result by directly using NUMBER_VAL(). Instead, the wrapper to\nuse is passed in as a macro parameter. For our\nexisting arithmetic operators, the result is a number, so we pass in the\nNUMBER_VAL macro.

\n\n
      }\n
vm.c
\nin run()
\nreplace 4 lines
\n
      case OP_ADD:      BINARY_OP(NUMBER_VAL, +); break;\n      case OP_SUBTRACT: BINARY_OP(NUMBER_VAL, -); break;\n      case OP_MULTIPLY: BINARY_OP(NUMBER_VAL, *); break;\n      case OP_DIVIDE:   BINARY_OP(NUMBER_VAL, /); break;\n
      case OP_NEGATE:\n
\n
vm.c, in run(), replace 4 lines
\n\n

Soon, I’ll show you why we made the wrapping macro an argument.

\n

18 . 4Two New Types

\n

All of our existing clox code is back in working order. Finally, it’s time to\nadd some new types. We’ve got a running numeric calculator that now does a\nnumber of pointless paranoid runtime type checks. We can represent other types\ninternally, but there’s no way for a user’s program to ever create a Value of\none of those types.

\n

Not until now, that is. We’ll start by adding compiler support for the three new\nliterals: true, false, and nil. They’re all pretty simple, so we’ll do all\nthree in a single batch.

\n

With number literals, we had to deal with the fact that there are billions of\npossible numeric values. We attended to that by storing the literal’s value in\nthe chunk’s constant table and emitting a bytecode instruction that simply\nloaded that constant. We could do the same thing for the new types. We’d store,\nsay, true, in the constant table, and use an OP_CONSTANT to read it out.

\n

But given that there are literally (heh) only three possible values we need to\nworry about with these new types, it’s gratuitous—and slow!—to waste a two-byte instruction and a constant\ntable entry on them. Instead, we’ll define three dedicated instructions to push\neach of these literals on the stack.

\n\n
  OP_CONSTANT,\n
chunk.h
\nin enum OpCode
\n
  OP_NIL,\n  OP_TRUE,\n  OP_FALSE,\n
  OP_ADD,\n
\n
chunk.h, in enum OpCode
\n\n

Our scanner already treats true, false, and nil as keywords, so we can\nskip right to the parser. With our table-based Pratt parser, we just need to\nslot parser functions into the rows associated with those keyword token types.\nWe’ll use the same function in all three slots. Here:

\n
  [TOKEN_ELSE]          = {NULL,     NULL,   PREC_NONE},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_FALSE]         = {literal,  NULL,   PREC_NONE},\n
  [TOKEN_FOR]           = {NULL,     NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n

Here:

\n
  [TOKEN_THIS]          = {NULL,     NULL,   PREC_NONE},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_TRUE]          = {literal,  NULL,   PREC_NONE},\n
  [TOKEN_VAR]           = {NULL,     NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n

And here:

\n
  [TOKEN_IF]            = {NULL,     NULL,   PREC_NONE},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_NIL]           = {literal,  NULL,   PREC_NONE},\n
  [TOKEN_OR]            = {NULL,     NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n

When the parser encounters false, nil, or true, in prefix position, it\ncalls this new parser function:

\n
compiler.c
\nadd after binary()
\n
static void literal() {\n  switch (parser.previous.type) {\n    case TOKEN_FALSE: emitByte(OP_FALSE); break;\n    case TOKEN_NIL: emitByte(OP_NIL); break;\n    case TOKEN_TRUE: emitByte(OP_TRUE); break;\n    default: return; // Unreachable.\n  }\n}\n
\n
compiler.c, add after binary()
\n\n

Since parsePrecedence() has already consumed the keyword token, all we need to\ndo is output the proper instruction. We figure that\nout based on the type of token we parsed. Our front end can now compile Boolean\nand nil literals to bytecode. Moving down the execution pipeline, we reach the\ninterpreter.

\n\n
      case OP_CONSTANT: {\n        Value constant = READ_CONSTANT();\n        push(constant);\n        break;\n      }\n
vm.c
\nin run()
\n
      case OP_NIL: push(NIL_VAL); break;\n      case OP_TRUE: push(BOOL_VAL(true)); break;\n      case OP_FALSE: push(BOOL_VAL(false)); break;\n
      case OP_ADD:      BINARY_OP(NUMBER_VAL, +); break;\n
\n
vm.c, in run()
\n\n

This is pretty self-explanatory. Each instruction summons the appropriate value\nand pushes it onto the stack. We shouldn’t forget our disassembler either.

\n
    case OP_CONSTANT:\n      return constantInstruction("OP_CONSTANT", chunk, offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_NIL:\n      return simpleInstruction("OP_NIL", offset);\n    case OP_TRUE:\n      return simpleInstruction("OP_TRUE", offset);\n    case OP_FALSE:\n      return simpleInstruction("OP_FALSE", offset);\n
    case OP_ADD:\n
\n
debug.c, in disassembleInstruction()
\n\n

With this in place, we can run this Earth-shattering program:

\n
true\n
\n

Except that when the interpreter tries to print the result, it blows up. We need\nto extend printValue() to handle the new types too:

\n
void printValue(Value value) {\n
value.c
\nin printValue()
\nreplace 1 line
\n
  switch (value.type) {\n    case VAL_BOOL:\n      printf(AS_BOOL(value) ? "true" : "false");\n      break;\n    case VAL_NIL: printf("nil"); break;\n    case VAL_NUMBER: printf("%g", AS_NUMBER(value)); break;\n  }\n
}\n
\n
value.c, in printValue(), replace 1 line
\n\n

There we go! Now we have some new types. They just aren’t very useful yet. Aside\nfrom the literals, you can’t really do anything with them. It will be a while\nbefore nil comes into play, but we can start putting Booleans to work in the\nlogical operators.

\n

18 . 4 . 1Logical not and falsiness

\n

The simplest logical operator is our old exclamatory friend unary not.

\n
print !true; // "false"\n
\n

This new operation gets a new instruction.

\n
  OP_DIVIDE,\n
chunk.h
\nin enum OpCode
\n
  OP_NOT,\n
  OP_NEGATE,\n
\n
chunk.h, in enum OpCode
\n\n

We can reuse the unary() parser function we wrote for unary negation to\ncompile a not expression. We just need to slot it into the parsing table.

\n
  [TOKEN_STAR]          = {NULL,     binary, PREC_FACTOR},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_BANG]          = {unary,    NULL,   PREC_NONE},\n
  [TOKEN_BANG_EQUAL]    = {NULL,     NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n

Because I knew we were going to do this, the unary() function already has a\nswitch on the token type to figure out which bytecode instruction to output. We\nmerely add another case.

\n
  switch (operatorType) {\n
compiler.c
\nin unary()
\n
    case TOKEN_BANG: emitByte(OP_NOT); break;\n
    case TOKEN_MINUS: emitByte(OP_NEGATE); break;\n    default: return; // Unreachable.\n  }\n
\n
compiler.c, in unary()
\n\n

That’s it for the front end. Let’s head over to the VM and conjure this\ninstruction into life.

\n
      case OP_DIVIDE:   BINARY_OP(NUMBER_VAL, /); break;\n
vm.c
\nin run()
\n
      case OP_NOT:\n        push(BOOL_VAL(isFalsey(pop())));\n        break;\n
      case OP_NEGATE:\n
\n
vm.c, in run()
\n\n

Like our previous unary operator, it pops the one operand, performs the\noperation, and pushes the result. And, as we did there, we have to worry about\ndynamic typing. Taking the logical not of true is easy, but there’s nothing\npreventing an unruly programmer from writing something like this:

\n
print !nil;\n
\n

For unary minus, we made it an error to negate anything that isn’t a number. But Lox, like most scripting languages, is more\npermissive when it comes to ! and other contexts where a Boolean is expected.\nThe rule for how other types are handled is called “falsiness”, and we implement\nit here:

\n\n
vm.c
\nadd after peek()
\n
static bool isFalsey(Value value) {\n  return IS_NIL(value) || (IS_BOOL(value) && !AS_BOOL(value));\n}\n
\n
vm.c, add after peek()
\n\n

Lox follows Ruby in that nil and false are falsey and every other value\nbehaves like true. We’ve got a new instruction we can generate, so we also\nneed to be able to ungenerate it in the disassembler.

\n
    case OP_DIVIDE:\n      return simpleInstruction("OP_DIVIDE", offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_NOT:\n      return simpleInstruction("OP_NOT", offset);\n
    case OP_NEGATE:\n
\n
debug.c, in disassembleInstruction()
\n\n

18 . 4 . 2Equality and comparison operators

\n

That wasn’t too bad. Let’s keep the momentum going and knock out the equality\nand comparison operators too: ==, !=, <, >, <=, and >=. That covers\nall of the operators that return Boolean results except the logical operators\nand and or. Since those need to short-circuit (basically do a little\ncontrol flow) we aren’t ready for them yet.

\n

Here are the new instructions for those operators:

\n
  OP_FALSE,\n
chunk.h
\nin enum OpCode
\n
  OP_EQUAL,\n  OP_GREATER,\n  OP_LESS,\n
  OP_ADD,\n
\n
chunk.h, in enum OpCode
\n\n

Wait, only three? What about !=, <=, and >=? We could create instructions\nfor those too. Honestly, the VM would execute faster if we did, so we should\ndo that if the goal is performance.

\n

But my main goal is to teach you about bytecode compilers. I want you to start\ninternalizing the idea that the bytecode instructions don’t need to closely\nfollow the user’s source code. The VM has total freedom to use whatever\ninstruction set and code sequences it wants as long as they have the right\nuser-visible behavior.

\n

The expression a != b has the same semantics as !(a == b), so the compiler\nis free to compile the former as if it were the latter. Instead of a dedicated\nOP_NOT_EQUAL instruction, it can output an OP_EQUAL followed by an OP_NOT.\nLikewise, a <= b is the same as !(a > b) and a >= b is !(a < b). Thus, we only need three new instructions.

\n\n

Over in the parser, though, we do have six new operators to slot into the parse\ntable. We use the same binary() parser function from before. Here’s the row\nfor !=:

\n
  [TOKEN_BANG]          = {unary,    NULL,   PREC_NONE},\n
compiler.c
\nreplace 1 line
\n
  [TOKEN_BANG_EQUAL]    = {NULL,     binary, PREC_EQUALITY},\n
  [TOKEN_EQUAL]         = {NULL,     NULL,   PREC_NONE},\n
\n
compiler.c, replace 1 line
\n\n

The remaining five operators are a little farther down in the table.

\n
  [TOKEN_EQUAL]         = {NULL,     NULL,   PREC_NONE},\n
compiler.c
\nreplace 5 lines
\n
  [TOKEN_EQUAL_EQUAL]   = {NULL,     binary, PREC_EQUALITY},\n  [TOKEN_GREATER]       = {NULL,     binary, PREC_COMPARISON},\n  [TOKEN_GREATER_EQUAL] = {NULL,     binary, PREC_COMPARISON},\n  [TOKEN_LESS]          = {NULL,     binary, PREC_COMPARISON},\n  [TOKEN_LESS_EQUAL]    = {NULL,     binary, PREC_COMPARISON},\n
  [TOKEN_IDENTIFIER]    = {NULL,     NULL,   PREC_NONE},\n
\n
compiler.c, replace 5 lines
\n\n

Inside binary() we already have a switch to generate the right bytecode for\neach token type. We add cases for the six new operators.

\n
  switch (operatorType) {\n
compiler.c
\nin binary()
\n
    case TOKEN_BANG_EQUAL:    emitBytes(OP_EQUAL, OP_NOT); break;\n    case TOKEN_EQUAL_EQUAL:   emitByte(OP_EQUAL); break;\n    case TOKEN_GREATER:       emitByte(OP_GREATER); break;\n    case TOKEN_GREATER_EQUAL: emitBytes(OP_LESS, OP_NOT); break;\n    case TOKEN_LESS:          emitByte(OP_LESS); break;\n    case TOKEN_LESS_EQUAL:    emitBytes(OP_GREATER, OP_NOT); break;\n
    case TOKEN_PLUS:          emitByte(OP_ADD); break;\n
\n
compiler.c, in binary()
\n\n

The ==, <, and > operators output a single instruction. The others output\na pair of instructions, one to evalute the inverse operation, and then an\nOP_NOT to flip the result. Six operators for the price of three instructions!

\n

That means over in the VM, our job is simpler. Equality is the most general\noperation.

\n
      case OP_FALSE: push(BOOL_VAL(false)); break;\n
vm.c
\nin run()
\n
      case OP_EQUAL: {\n        Value b = pop();\n        Value a = pop();\n        push(BOOL_VAL(valuesEqual(a, b)));\n        break;\n      }\n
      case OP_ADD:      BINARY_OP(NUMBER_VAL, +); break;\n
\n
vm.c, in run()
\n\n

You can evaluate == on any pair of objects, even objects of different types.\nThere’s enough complexity that it makes sense to shunt that logic over to a\nseparate function. That function always returns a C bool, so we can safely\nwrap the result in a BOOL_VAL. The function relates to Values, so it lives\nover in the “value” module.

\n
} ValueArray;\n\n
value.h
\nadd after struct ValueArray
\n
bool valuesEqual(Value a, Value b);\n
void initValueArray(ValueArray* array);\n
\n
value.h, add after struct ValueArray
\n\n

And here’s the implementation:

\n
value.c
\nadd after printValue()
\n
bool valuesEqual(Value a, Value b) {\n  if (a.type != b.type) return false;\n  switch (a.type) {\n    case VAL_BOOL:   return AS_BOOL(a) == AS_BOOL(b);\n    case VAL_NIL:    return true;\n    case VAL_NUMBER: return AS_NUMBER(a) == AS_NUMBER(b);\n    default:         return false; // Unreachable.\n  }\n}\n
\n
value.c, add after printValue()
\n\n

First, we check the types. If the Values have different types, they are definitely not equal. Otherwise,\nwe unwrap the two Values and compare them directly.

\n\n

For each value type, we have a separate case that handles comparing the value\nitself. Given how similar the cases are, you might wonder why we can’t simply\nmemcmp() the two Value structs and be done with it. The problem is that\nbecause of padding and different-sized union fields, a Value contains unused\nbits. C gives no guarantee about what is in those, so it’s possible that two\nequal Values actually differ in memory that isn’t used.

\"The\n

(You wouldn’t believe how much pain I went through before learning this fact.)

\n

Anyway, as we add more types to clox, this function will grow new cases. For\nnow, these three are sufficient. The other comparison operators are easier since\nthey work only on numbers.

\n
        push(BOOL_VAL(valuesEqual(a, b)));\n        break;\n      }\n
vm.c
\nin run()
\n
      case OP_GREATER:  BINARY_OP(BOOL_VAL, >); break;\n      case OP_LESS:     BINARY_OP(BOOL_VAL, <); break;\n
      case OP_ADD:      BINARY_OP(NUMBER_VAL, +); break;\n
\n
vm.c, in run()
\n\n

We already extended the BINARY_OP macro to handle operators that return\nnon-numeric types. Now we get to use that. We pass in BOOL_VAL since the\nresult value type is Boolean. Otherwise, it’s no different from plus or minus.

\n

As always, the coda to today’s aria is disassembling the new instructions.

\n
    case OP_FALSE:\n      return simpleInstruction("OP_FALSE", offset);\n
debug.c
\nin disassembleInstruction()
\n
    case OP_EQUAL:\n      return simpleInstruction("OP_EQUAL", offset);\n    case OP_GREATER:\n      return simpleInstruction("OP_GREATER", offset);\n    case OP_LESS:\n      return simpleInstruction("OP_LESS", offset);\n
    case OP_ADD:\n
\n
debug.c, in disassembleInstruction()
\n\n

With that, our numeric calculator has become something closer to a general\nexpression evaluator. Fire up clox and type in:

\n
!(5 - 4 > 3 * 2 == !nil)\n
\n

OK, I’ll admit that’s maybe not the most useful expression, but we’re making\nprogress. We have one missing built-in type with its own literal form: strings.\nThose are much more complex because strings can vary in size. That tiny\ndifference turns out to have implications so large that we give strings their\nvery own chapter.

\n
\n

Challenges

\n
    \n
  1. \n

    We could reduce our binary operators even further than we did here. Which\nother instructions can you eliminate, and how would the compiler cope with\ntheir absence?

    \n
    2. \n
  2. \n

    Conversely, we can improve the speed of our bytecode VM by adding more\nspecific instructions that correspond to higher-level operations. What\ninstructions would you define to speed up the kind of user code we added\nsupport for in this chapter?

    \n
    3. \n
\n
\n\n\n
\n\n
\n\n\n" }, { "path": "site/welcome.html", "content": "\n\n\n\nWelcome · Crafting Interpreters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
\n\n
\n\n
\n\n
I
\n

Welcome

\n\n

This may be the beginning of a grand adventure. Programming languages encompass\na huge space to explore and play in. Plenty of room for your own creations to\nshare with others or just enjoy yourself. Brilliant computer scientists and\nsoftware engineers have spent entire careers traversing this land without ever\nreaching the end. If this book is your first entry into the country, welcome.

\n

The pages of this book give you a guided tour through some of the world of\nlanguages. But before we strap on our hiking boots and venture out, we should\nfamiliarize ourselves with the territory. The chapters in this part introduce\nyou to the basic concepts used by programming languages and how those concepts\nare organized.

\n

We will also get acquainted with Lox, the language we’ll spend the rest of the\nbook implementing (twice).

\n\n\n
\n\n
\n\n\n" }, { "path": "test/assignment/associativity.lox", "content": "var a = \"a\";\nvar b = \"b\";\nvar c = \"c\";\n\n// Assignment is right-associative.\na = b = c;\nprint a; // expect: c\nprint b; // expect: c\nprint c; // expect: c\n" }, { "path": "test/assignment/global.lox", "content": "var a = \"before\";\nprint a; // expect: before\n\na = \"after\";\nprint a; // expect: after\n\nprint a = \"arg\"; // expect: arg\nprint a; // expect: arg\n" }, { "path": "test/assignment/grouping.lox", "content": "var a = \"a\";\n(a) = \"value\"; // Error at '=': Invalid assignment target.\n" }, { "path": "test/assignment/infix_operator.lox", "content": "var a = \"a\";\nvar b = \"b\";\na + b = \"value\"; // Error at '=': Invalid assignment target.\n" }, { "path": "test/assignment/local.lox", "content": "{\n var a = \"before\";\n print a; // expect: before\n\n a = \"after\";\n print a; // expect: after\n\n print a = \"arg\"; // expect: arg\n print a; // expect: arg\n}\n" }, { "path": "test/assignment/prefix_operator.lox", "content": "var a = \"a\";\n!a = \"value\"; // Error at '=': Invalid assignment target.\n" }, { "path": "test/assignment/syntax.lox", "content": "// Assignment on RHS of variable.\nvar a = \"before\";\nvar c = a = \"var\";\nprint a; // expect: var\nprint c; // expect: var\n" }, { "path": "test/assignment/to_this.lox", "content": "class Foo {\n Foo() {\n this = \"value\"; // Error at '=': Invalid assignment target.\n }\n}\n\nFoo();\n" }, { "path": "test/assignment/undefined.lox", "content": "unknown = \"what\"; // expect runtime error: Undefined variable 'unknown'.\n" }, { "path": "test/benchmark/binary_trees.lox", "content": "class Tree {\n init(item, depth) {\n this.item = item;\n this.depth = depth;\n if (depth > 0) {\n var item2 = item + item;\n depth = depth - 1;\n this.left = Tree(item2 - 1, depth);\n this.right = Tree(item2, depth);\n } else {\n this.left = nil;\n this.right = nil;\n }\n }\n\n check() {\n if (this.left == nil) {\n return this.item;\n }\n\n return this.item + this.left.check() - this.right.check();\n }\n}\n\nvar minDepth = 4;\nvar maxDepth = 14;\nvar stretchDepth = maxDepth + 1;\n\nvar start = clock();\n\nprint \"stretch tree of depth:\";\nprint stretchDepth;\nprint \"check:\";\nprint Tree(0, stretchDepth).check();\n\nvar longLivedTree = Tree(0, maxDepth);\n\n// iterations = 2 ** maxDepth\nvar iterations = 1;\nvar d = 0;\nwhile (d < maxDepth) {\n iterations = iterations * 2;\n d = d + 1;\n}\n\nvar depth = minDepth;\nwhile (depth < stretchDepth) {\n var check = 0;\n var i = 1;\n while (i <= iterations) {\n check = check + Tree(i, depth).check() + Tree(-i, depth).check();\n i = i + 1;\n }\n\n print \"num trees:\";\n print iterations * 2;\n print \"depth:\";\n print depth;\n print \"check:\";\n print check;\n\n iterations = iterations / 4;\n depth = depth + 2;\n}\n\nprint \"long lived tree of depth:\";\nprint maxDepth;\nprint \"check:\";\nprint longLivedTree.check();\nprint \"elapsed:\";\nprint clock() - start;\n" }, { "path": "test/benchmark/equality.lox", "content": "var i = 0;\n\nvar loopStart = clock();\n\nwhile (i < 10000000) {\n i = i + 1;\n\n 1; 1; 1; 2; 1; nil; 1; \"str\"; 1; true;\n nil; nil; nil; 1; nil; \"str\"; nil; true;\n true; true; true; 1; true; false; true; \"str\"; true; nil;\n \"str\"; \"str\"; \"str\"; \"stru\"; \"str\"; 1; \"str\"; nil; \"str\"; true;\n}\n\nvar loopTime = clock() - loopStart;\n\nvar start = clock();\n\ni = 0;\nwhile (i < 10000000) {\n i = i + 1;\n\n 1 == 1; 1 == 2; 1 == nil; 1 == \"str\"; 1 == true;\n nil == nil; nil == 1; nil == \"str\"; nil == true;\n true == true; true == 1; true == false; true == \"str\"; true == nil;\n \"str\" == \"str\"; \"str\" == \"stru\"; \"str\" == 1; \"str\" == nil; \"str\" == true;\n}\n\nvar elapsed = clock() - start;\nprint \"loop\";\nprint loopTime;\nprint \"elapsed\";\nprint elapsed;\nprint \"equals\";\nprint elapsed - loopTime;\n" }, { "path": "test/benchmark/fib.lox", "content": "fun fib(n) {\n if (n < 2) return n;\n return fib(n - 2) + fib(n - 1);\n}\n\nvar start = clock();\nprint fib(35) == 9227465;\nprint clock() - start;\n" }, { "path": "test/benchmark/instantiation.lox", "content": "// This benchmark stresses instance creation and initializer calling.\n\nclass Foo {\n init() {}\n}\n\nvar start = clock();\nvar i = 0;\nwhile (i < 500000) {\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n Foo();\n i = i + 1;\n}\n\nprint clock() - start;\n" }, { "path": "test/benchmark/invocation.lox", "content": "// This benchmark stresses just method invocation.\n\nclass Foo {\n method0() {}\n method1() {}\n method2() {}\n method3() {}\n method4() {}\n method5() {}\n method6() {}\n method7() {}\n method8() {}\n method9() {}\n method10() {}\n method11() {}\n method12() {}\n method13() {}\n method14() {}\n method15() {}\n method16() {}\n method17() {}\n method18() {}\n method19() {}\n method20() {}\n method21() {}\n method22() {}\n method23() {}\n method24() {}\n method25() {}\n method26() {}\n method27() {}\n method28() {}\n method29() {}\n}\n\nvar foo = Foo();\nvar start = clock();\nvar i = 0;\nwhile (i < 500000) {\n foo.method0();\n foo.method1();\n foo.method2();\n foo.method3();\n foo.method4();\n foo.method5();\n foo.method6();\n foo.method7();\n foo.method8();\n foo.method9();\n foo.method10();\n foo.method11();\n foo.method12();\n foo.method13();\n foo.method14();\n foo.method15();\n foo.method16();\n foo.method17();\n foo.method18();\n foo.method19();\n foo.method20();\n foo.method21();\n foo.method22();\n foo.method23();\n foo.method24();\n foo.method25();\n foo.method26();\n foo.method27();\n foo.method28();\n foo.method29();\n i = i + 1;\n}\n\nprint clock() - start;\n" }, { "path": "test/benchmark/method_call.lox", "content": "class Toggle {\n init(startState) {\n this.state = startState;\n }\n\n value() { return this.state; }\n\n activate() {\n this.state = !this.state;\n return this;\n }\n}\n\nclass NthToggle < Toggle {\n init(startState, maxCounter) {\n super.init(startState);\n this.countMax = maxCounter;\n this.count = 0;\n }\n\n activate() {\n this.count = this.count + 1;\n if (this.count >= this.countMax) {\n super.activate();\n this.count = 0;\n }\n\n return this;\n }\n}\n\nvar start = clock();\nvar n = 100000;\nvar val = true;\nvar toggle = Toggle(val);\n\nfor (var i = 0; i < n; i = i + 1) {\n val = toggle.activate().value();\n val = toggle.activate().value();\n val = toggle.activate().value();\n val = toggle.activate().value();\n val = toggle.activate().value();\n val = toggle.activate().value();\n val = toggle.activate().value();\n val = toggle.activate().value();\n val = toggle.activate().value();\n val = toggle.activate().value();\n}\n\nprint toggle.value();\n\nval = true;\nvar ntoggle = NthToggle(val, 3);\n\nfor (var i = 0; i < n; i = i + 1) {\n val = ntoggle.activate().value();\n val = ntoggle.activate().value();\n val = ntoggle.activate().value();\n val = ntoggle.activate().value();\n val = ntoggle.activate().value();\n val = ntoggle.activate().value();\n val = ntoggle.activate().value();\n val = ntoggle.activate().value();\n val = ntoggle.activate().value();\n val = ntoggle.activate().value();\n}\n\nprint ntoggle.value();\nprint clock() - start;\n" }, { "path": "test/benchmark/properties.lox", "content": "// This benchmark stresses both field and method lookup.\n\nclass Foo {\n init() {\n this.field0 = 1;\n this.field1 = 1;\n this.field2 = 1;\n this.field3 = 1;\n this.field4 = 1;\n this.field5 = 1;\n this.field6 = 1;\n this.field7 = 1;\n this.field8 = 1;\n this.field9 = 1;\n this.field10 = 1;\n this.field11 = 1;\n this.field12 = 1;\n this.field13 = 1;\n this.field14 = 1;\n this.field15 = 1;\n this.field16 = 1;\n this.field17 = 1;\n this.field18 = 1;\n this.field19 = 1;\n this.field20 = 1;\n this.field21 = 1;\n this.field22 = 1;\n this.field23 = 1;\n this.field24 = 1;\n this.field25 = 1;\n this.field26 = 1;\n this.field27 = 1;\n this.field28 = 1;\n this.field29 = 1;\n }\n\n method0() { return this.field0; }\n method1() { return this.field1; }\n method2() { return this.field2; }\n method3() { return this.field3; }\n method4() { return this.field4; }\n method5() { return this.field5; }\n method6() { return this.field6; }\n method7() { return this.field7; }\n method8() { return this.field8; }\n method9() { return this.field9; }\n method10() { return this.field10; }\n method11() { return this.field11; }\n method12() { return this.field12; }\n method13() { return this.field13; }\n method14() { return this.field14; }\n method15() { return this.field15; }\n method16() { return this.field16; }\n method17() { return this.field17; }\n method18() { return this.field18; }\n method19() { return this.field19; }\n method20() { return this.field20; }\n method21() { return this.field21; }\n method22() { return this.field22; }\n method23() { return this.field23; }\n method24() { return this.field24; }\n method25() { return this.field25; }\n method26() { return this.field26; }\n method27() { return this.field27; }\n method28() { return this.field28; }\n method29() { return this.field29; }\n}\n\nvar foo = Foo();\nvar start = clock();\nvar i = 0;\nwhile (i < 500000) {\n foo.method0();\n foo.method1();\n foo.method2();\n foo.method3();\n foo.method4();\n foo.method5();\n foo.method6();\n foo.method7();\n foo.method8();\n foo.method9();\n foo.method10();\n foo.method11();\n foo.method12();\n foo.method13();\n foo.method14();\n foo.method15();\n foo.method16();\n foo.method17();\n foo.method18();\n foo.method19();\n foo.method20();\n foo.method21();\n foo.method22();\n foo.method23();\n foo.method24();\n foo.method25();\n foo.method26();\n foo.method27();\n foo.method28();\n foo.method29();\n i = i + 1;\n}\n\nprint clock() - start;\n" }, { "path": "test/benchmark/string_equality.lox", "content": "var a1 = \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa1\";\nvar a2 = \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa2\";\nvar a3 = \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa3\";\nvar a4 = \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa4\";\nvar a5 = \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa5\";\nvar a6 = \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa6\";\nvar a7 = \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa7\";\nvar a8 = \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8\";\n\nvar i = 0;\n\nvar loopStart = clock();\n\nwhile (i < 100000) {\n i = i + 1;\n\n a1; a1; a1; a2; a1; a3; a1; a4; a1; a5; a1; a6; a1; a7; a1; a8;\n a2; a1; a2; a2; a2; a3; a2; a4; a2; a5; a2; a6; a2; a7; a2; a8;\n a3; a1; a3; a2; a3; a3; a3; a4; a3; a5; a3; a6; a3; a7; a3; a8;\n a4; a1; a4; a2; a4; a3; a4; a4; a4; a5; a4; a6; a4; a7; a4; a8;\n a5; a1; a5; a2; a5; a3; a5; a4; a5; a5; a5; a6; a5; a7; a5; a8;\n a6; a1; a6; a2; a6; a3; a6; a4; a6; a5; a6; a6; a6; a7; a6; a8;\n a7; a1; a7; a2; a7; a3; a7; a4; a7; a5; a7; a6; a7; a7; a7; a8;\n a8; a1; a8; a2; a8; a3; a8; a4; a8; a5; a8; a6; a8; a7; a8; a8;\n\n a1; a1; a1; a2; a1; a3; a1; a4; a1; a5; a1; a6; a1; a7; a1; a8;\n a2; a1; a2; a2; a2; a3; a2; a4; a2; a5; a2; a6; a2; a7; a2; a8;\n a3; a1; a3; a2; a3; a3; a3; a4; a3; a5; a3; a6; a3; a7; a3; a8;\n a4; a1; a4; a2; a4; a3; a4; a4; a4; a5; a4; a6; a4; a7; a4; a8;\n a5; a1; a5; a2; a5; a3; a5; a4; a5; a5; a5; a6; a5; a7; a5; a8;\n a6; a1; a6; a2; a6; a3; a6; a4; a6; a5; a6; a6; a6; a7; a6; a8;\n a7; a1; a7; a2; a7; a3; a7; a4; a7; a5; a7; a6; a7; a7; a7; a8;\n a8; a1; a8; a2; a8; a3; a8; a4; a8; a5; a8; a6; a8; a7; a8; a8;\n\n a1; a1; a1; a2; a1; a3; a1; a4; a1; a5; a1; a6; a1; a7; a1; a8;\n a2; a1; a2; a2; a2; a3; a2; a4; a2; a5; a2; a6; a2; a7; a2; a8;\n a3; a1; a3; a2; a3; a3; a3; a4; a3; a5; a3; a6; a3; a7; a3; a8;\n a4; a1; a4; a2; a4; a3; a4; a4; a4; a5; a4; a6; a4; a7; a4; a8;\n a5; a1; a5; a2; a5; a3; a5; a4; a5; a5; a5; a6; a5; a7; a5; a8;\n a6; a1; a6; a2; a6; a3; a6; a4; a6; a5; a6; a6; a6; a7; a6; a8;\n a7; a1; a7; a2; a7; a3; a7; a4; a7; a5; a7; a6; a7; a7; a7; a8;\n a8; a1; a8; a2; a8; a3; a8; a4; a8; a5; a8; a6; a8; a7; a8; a8;\n\n a1; a1; a1; a2; a1; a3; a1; a4; a1; a5; a1; a6; a1; a7; a1; a8;\n a2; a1; a2; a2; a2; a3; a2; a4; a2; a5; a2; a6; a2; a7; a2; a8;\n a3; a1; a3; a2; a3; a3; a3; a4; a3; a5; a3; a6; a3; a7; a3; a8;\n a4; a1; a4; a2; a4; a3; a4; a4; a4; a5; a4; a6; a4; a7; a4; a8;\n a5; a1; a5; a2; a5; a3; a5; a4; a5; a5; a5; a6; a5; a7; a5; a8;\n a6; a1; a6; a2; a6; a3; a6; a4; a6; a5; a6; a6; a6; a7; a6; a8;\n a7; a1; a7; a2; a7; a3; a7; a4; a7; a5; a7; a6; a7; a7; a7; a8;\n a8; a1; a8; a2; a8; a3; a8; a4; a8; a5; a8; a6; a8; a7; a8; a8;\n\n a1; a1; a1; a2; a1; a3; a1; a4; a1; a5; a1; a6; a1; a7; a1; a8;\n a2; a1; a2; a2; a2; a3; a2; a4; a2; a5; a2; a6; a2; a7; a2; a8;\n a3; a1; a3; a2; a3; a3; a3; a4; a3; a5; a3; a6; a3; a7; a3; a8;\n a4; a1; a4; a2; a4; a3; a4; a4; a4; a5; a4; a6; a4; a7; a4; a8;\n a5; a1; a5; a2; a5; a3; a5; a4; a5; a5; a5; a6; a5; a7; a5; a8;\n a6; a1; a6; a2; a6; a3; a6; a4; a6; a5; a6; a6; a6; a7; a6; a8;\n a7; a1; a7; a2; a7; a3; a7; a4; a7; a5; a7; a6; a7; a7; a7; a8;\n a8; a1; a8; a2; a8; a3; a8; a4; a8; a5; a8; a6; a8; a7; a8; a8;\n\n a1; a1; a1; a2; a1; a3; a1; a4; a1; a5; a1; a6; a1; a7; a1; a8;\n a2; a1; a2; a2; a2; a3; a2; a4; a2; a5; a2; a6; a2; a7; a2; a8;\n a3; a1; a3; a2; a3; a3; a3; a4; a3; a5; a3; a6; a3; a7; a3; a8;\n a4; a1; a4; a2; a4; a3; a4; a4; a4; a5; a4; a6; a4; a7; a4; a8;\n a5; a1; a5; a2; a5; a3; a5; a4; a5; a5; a5; a6; a5; a7; a5; a8;\n a6; a1; a6; a2; a6; a3; a6; a4; a6; a5; a6; a6; a6; a7; a6; a8;\n a7; a1; a7; a2; a7; a3; a7; a4; a7; a5; a7; a6; a7; a7; a7; a8;\n a8; a1; a8; a2; a8; a3; a8; a4; a8; a5; a8; a6; a8; a7; a8; a8;\n\n a1; a1; a1; a2; a1; a3; a1; a4; a1; a5; a1; a6; a1; a7; a1; a8;\n a2; a1; a2; a2; a2; a3; a2; a4; a2; a5; a2; a6; a2; a7; a2; a8;\n a3; a1; a3; a2; a3; a3; a3; a4; a3; a5; a3; a6; a3; a7; a3; a8;\n a4; a1; a4; a2; a4; a3; a4; a4; a4; a5; a4; a6; a4; a7; a4; a8;\n a5; a1; a5; a2; a5; a3; a5; a4; a5; a5; a5; a6; a5; a7; a5; a8;\n a6; a1; a6; a2; a6; a3; a6; a4; a6; a5; a6; a6; a6; a7; a6; a8;\n a7; a1; a7; a2; a7; a3; a7; a4; a7; a5; a7; a6; a7; a7; a7; a8;\n a8; a1; a8; a2; a8; a3; a8; a4; a8; a5; a8; a6; a8; a7; a8; a8;\n\n a1; a1; a1; a2; a1; a3; a1; a4; a1; a5; a1; a6; a1; a7; a1; a8;\n a2; a1; a2; a2; a2; a3; a2; a4; a2; a5; a2; a6; a2; a7; a2; a8;\n a3; a1; a3; a2; a3; a3; a3; a4; a3; a5; a3; a6; a3; a7; a3; a8;\n a4; a1; a4; a2; a4; a3; a4; a4; a4; a5; a4; a6; a4; a7; a4; a8;\n a5; a1; a5; a2; a5; a3; a5; a4; a5; a5; a5; a6; a5; a7; a5; a8;\n a6; a1; a6; a2; a6; a3; a6; a4; a6; a5; a6; a6; a6; a7; a6; a8;\n a7; a1; a7; a2; a7; a3; a7; a4; a7; a5; a7; a6; a7; a7; a7; a8;\n a8; a1; a8; a2; a8; a3; a8; a4; a8; a5; a8; a6; a8; a7; a8; a8;\n\n a1; a1; a1; a2; a1; a3; a1; a4; a1; a5; a1; a6; a1; a7; a1; a8;\n a2; a1; a2; a2; a2; a3; a2; a4; a2; a5; a2; a6; a2; a7; a2; a8;\n a3; a1; a3; a2; a3; a3; a3; a4; a3; a5; a3; a6; a3; a7; a3; a8;\n a4; a1; a4; a2; a4; a3; a4; a4; a4; a5; a4; a6; a4; a7; a4; a8;\n a5; a1; a5; a2; a5; a3; a5; a4; a5; a5; a5; a6; a5; a7; a5; a8;\n a6; a1; a6; a2; a6; a3; a6; a4; a6; a5; a6; a6; a6; a7; a6; a8;\n a7; a1; a7; a2; a7; a3; a7; a4; a7; a5; a7; a6; a7; a7; a7; a8;\n a8; a1; a8; a2; a8; a3; a8; a4; a8; a5; a8; a6; a8; a7; a8; a8;\n\n a1; a1; a1; a2; a1; a3; a1; a4; a1; a5; a1; a6; a1; a7; a1; a8;\n a2; a1; a2; a2; a2; a3; a2; a4; a2; a5; a2; a6; a2; a7; a2; a8;\n a3; a1; a3; a2; a3; a3; a3; a4; a3; a5; a3; a6; a3; a7; a3; a8;\n a4; a1; a4; a2; a4; a3; a4; a4; a4; a5; a4; a6; a4; a7; a4; a8;\n a5; a1; a5; a2; a5; a3; a5; a4; a5; a5; a5; a6; a5; a7; a5; a8;\n a6; a1; a6; a2; a6; a3; a6; a4; a6; a5; a6; a6; a6; a7; a6; a8;\n a7; a1; a7; a2; a7; a3; a7; a4; a7; a5; a7; a6; a7; a7; a7; a8;\n a8; a1; a8; a2; a8; a3; a8; a4; a8; a5; a8; a6; a8; a7; a8; a8;\n}\n\nvar loopTime = clock() - loopStart;\n\nvar start = clock();\n\ni = 0;\nwhile (i < 100000) {\n i = i + 1;\n\n // 1 == 1; 1 == 2; 1 == nil; 1 == \"str\"; 1 == true;\n // nil == nil; nil == 1; nil == \"str\"; nil == true;\n // true == true; true == 1; true == false; true == \"str\"; true == nil;\n // \"str\" == \"str\"; \"str\" == \"stru\"; \"str\" == 1; \"str\" == nil; \"str\" == true;\n\n a1 == a1; a1 == a2; a1 == a3; a1 == a4; a1 == a5; a1 == a6; a1 == a7; a1 == a8;\n a2 == a1; a2 == a2; a2 == a3; a2 == a4; a2 == a5; a2 == a6; a2 == a7; a2 == a8;\n a3 == a1; a3 == a2; a3 == a3; a3 == a4; a3 == a5; a3 == a6; a3 == a7; a3 == a8;\n a4 == a1; a4 == a2; a4 == a3; a4 == a4; a4 == a5; a4 == a6; a4 == a7; a4 == a8;\n a5 == a1; a5 == a2; a5 == a3; a5 == a4; a5 == a5; a5 == a6; a5 == a7; a5 == a8;\n a6 == a1; a6 == a2; a6 == a3; a6 == a4; a6 == a5; a6 == a6; a6 == a7; a6 == a8;\n a7 == a1; a7 == a2; a7 == a3; a7 == a4; a7 == a5; a7 == a6; a7 == a7; a7 == a8;\n a8 == a1; a8 == a2; a8 == a3; a8 == a4; a8 == a5; a8 == a6; a8 == a7; a8 == a8;\n\n a1 == a1; a1 == a2; a1 == a3; a1 == a4; a1 == a5; a1 == a6; a1 == a7; a1 == a8;\n a2 == a1; a2 == a2; a2 == a3; a2 == a4; a2 == a5; a2 == a6; a2 == a7; a2 == a8;\n a3 == a1; a3 == a2; a3 == a3; a3 == a4; a3 == a5; a3 == a6; a3 == a7; a3 == a8;\n a4 == a1; a4 == a2; a4 == a3; a4 == a4; a4 == a5; a4 == a6; a4 == a7; a4 == a8;\n a5 == a1; a5 == a2; a5 == a3; a5 == a4; a5 == a5; a5 == a6; a5 == a7; a5 == a8;\n a6 == a1; a6 == a2; a6 == a3; a6 == a4; a6 == a5; a6 == a6; a6 == a7; a6 == a8;\n a7 == a1; a7 == a2; a7 == a3; a7 == a4; a7 == a5; a7 == a6; a7 == a7; a7 == a8;\n a8 == a1; a8 == a2; a8 == a3; a8 == a4; a8 == a5; a8 == a6; a8 == a7; a8 == a8;\n\n a1 == a1; a1 == a2; a1 == a3; a1 == a4; a1 == a5; a1 == a6; a1 == a7; a1 == a8;\n a2 == a1; a2 == a2; a2 == a3; a2 == a4; a2 == a5; a2 == a6; a2 == a7; a2 == a8;\n a3 == a1; a3 == a2; a3 == a3; a3 == a4; a3 == a5; a3 == a6; a3 == a7; a3 == a8;\n a4 == a1; a4 == a2; a4 == a3; a4 == a4; a4 == a5; a4 == a6; a4 == a7; a4 == a8;\n a5 == a1; a5 == a2; a5 == a3; a5 == a4; a5 == a5; a5 == a6; a5 == a7; a5 == a8;\n a6 == a1; a6 == a2; a6 == a3; a6 == a4; a6 == a5; a6 == a6; a6 == a7; a6 == a8;\n a7 == a1; a7 == a2; a7 == a3; a7 == a4; a7 == a5; a7 == a6; a7 == a7; a7 == a8;\n a8 == a1; a8 == a2; a8 == a3; a8 == a4; a8 == a5; a8 == a6; a8 == a7; a8 == a8;\n\n a1 == a1; a1 == a2; a1 == a3; a1 == a4; a1 == a5; a1 == a6; a1 == a7; a1 == a8;\n a2 == a1; a2 == a2; a2 == a3; a2 == a4; a2 == a5; a2 == a6; a2 == a7; a2 == a8;\n a3 == a1; a3 == a2; a3 == a3; a3 == a4; a3 == a5; a3 == a6; a3 == a7; a3 == a8;\n a4 == a1; a4 == a2; a4 == a3; a4 == a4; a4 == a5; a4 == a6; a4 == a7; a4 == a8;\n a5 == a1; a5 == a2; a5 == a3; a5 == a4; a5 == a5; a5 == a6; a5 == a7; a5 == a8;\n a6 == a1; a6 == a2; a6 == a3; a6 == a4; a6 == a5; a6 == a6; a6 == a7; a6 == a8;\n a7 == a1; a7 == a2; a7 == a3; a7 == a4; a7 == a5; a7 == a6; a7 == a7; a7 == a8;\n a8 == a1; a8 == a2; a8 == a3; a8 == a4; a8 == a5; a8 == a6; a8 == a7; a8 == a8;\n\n a1 == a1; a1 == a2; a1 == a3; a1 == a4; a1 == a5; a1 == a6; a1 == a7; a1 == a8;\n a2 == a1; a2 == a2; a2 == a3; a2 == a4; a2 == a5; a2 == a6; a2 == a7; a2 == a8;\n a3 == a1; a3 == a2; a3 == a3; a3 == a4; a3 == a5; a3 == a6; a3 == a7; a3 == a8;\n a4 == a1; a4 == a2; a4 == a3; a4 == a4; a4 == a5; a4 == a6; a4 == a7; a4 == a8;\n a5 == a1; a5 == a2; a5 == a3; a5 == a4; a5 == a5; a5 == a6; a5 == a7; a5 == a8;\n a6 == a1; a6 == a2; a6 == a3; a6 == a4; a6 == a5; a6 == a6; a6 == a7; a6 == a8;\n a7 == a1; a7 == a2; a7 == a3; a7 == a4; a7 == a5; a7 == a6; a7 == a7; a7 == a8;\n a8 == a1; a8 == a2; a8 == a3; a8 == a4; a8 == a5; a8 == a6; a8 == a7; a8 == a8;\n\n a1 == a1; a1 == a2; a1 == a3; a1 == a4; a1 == a5; a1 == a6; a1 == a7; a1 == a8;\n a2 == a1; a2 == a2; a2 == a3; a2 == a4; a2 == a5; a2 == a6; a2 == a7; a2 == a8;\n a3 == a1; a3 == a2; a3 == a3; a3 == a4; a3 == a5; a3 == a6; a3 == a7; a3 == a8;\n a4 == a1; a4 == a2; a4 == a3; a4 == a4; a4 == a5; a4 == a6; a4 == a7; a4 == a8;\n a5 == a1; a5 == a2; a5 == a3; a5 == a4; a5 == a5; a5 == a6; a5 == a7; a5 == a8;\n a6 == a1; a6 == a2; a6 == a3; a6 == a4; a6 == a5; a6 == a6; a6 == a7; a6 == a8;\n a7 == a1; a7 == a2; a7 == a3; a7 == a4; a7 == a5; a7 == a6; a7 == a7; a7 == a8;\n a8 == a1; a8 == a2; a8 == a3; a8 == a4; a8 == a5; a8 == a6; a8 == a7; a8 == a8;\n\n a1 == a1; a1 == a2; a1 == a3; a1 == a4; a1 == a5; a1 == a6; a1 == a7; a1 == a8;\n a2 == a1; a2 == a2; a2 == a3; a2 == a4; a2 == a5; a2 == a6; a2 == a7; a2 == a8;\n a3 == a1; a3 == a2; a3 == a3; a3 == a4; a3 == a5; a3 == a6; a3 == a7; a3 == a8;\n a4 == a1; a4 == a2; a4 == a3; a4 == a4; a4 == a5; a4 == a6; a4 == a7; a4 == a8;\n a5 == a1; a5 == a2; a5 == a3; a5 == a4; a5 == a5; a5 == a6; a5 == a7; a5 == a8;\n a6 == a1; a6 == a2; a6 == a3; a6 == a4; a6 == a5; a6 == a6; a6 == a7; a6 == a8;\n a7 == a1; a7 == a2; a7 == a3; a7 == a4; a7 == a5; a7 == a6; a7 == a7; a7 == a8;\n a8 == a1; a8 == a2; a8 == a3; a8 == a4; a8 == a5; a8 == a6; a8 == a7; a8 == a8;\n\n a1 == a1; a1 == a2; a1 == a3; a1 == a4; a1 == a5; a1 == a6; a1 == a7; a1 == a8;\n a2 == a1; a2 == a2; a2 == a3; a2 == a4; a2 == a5; a2 == a6; a2 == a7; a2 == a8;\n a3 == a1; a3 == a2; a3 == a3; a3 == a4; a3 == a5; a3 == a6; a3 == a7; a3 == a8;\n a4 == a1; a4 == a2; a4 == a3; a4 == a4; a4 == a5; a4 == a6; a4 == a7; a4 == a8;\n a5 == a1; a5 == a2; a5 == a3; a5 == a4; a5 == a5; a5 == a6; a5 == a7; a5 == a8;\n a6 == a1; a6 == a2; a6 == a3; a6 == a4; a6 == a5; a6 == a6; a6 == a7; a6 == a8;\n a7 == a1; a7 == a2; a7 == a3; a7 == a4; a7 == a5; a7 == a6; a7 == a7; a7 == a8;\n a8 == a1; a8 == a2; a8 == a3; a8 == a4; a8 == a5; a8 == a6; a8 == a7; a8 == a8;\n\n a1 == a1; a1 == a2; a1 == a3; a1 == a4; a1 == a5; a1 == a6; a1 == a7; a1 == a8;\n a2 == a1; a2 == a2; a2 == a3; a2 == a4; a2 == a5; a2 == a6; a2 == a7; a2 == a8;\n a3 == a1; a3 == a2; a3 == a3; a3 == a4; a3 == a5; a3 == a6; a3 == a7; a3 == a8;\n a4 == a1; a4 == a2; a4 == a3; a4 == a4; a4 == a5; a4 == a6; a4 == a7; a4 == a8;\n a5 == a1; a5 == a2; a5 == a3; a5 == a4; a5 == a5; a5 == a6; a5 == a7; a5 == a8;\n a6 == a1; a6 == a2; a6 == a3; a6 == a4; a6 == a5; a6 == a6; a6 == a7; a6 == a8;\n a7 == a1; a7 == a2; a7 == a3; a7 == a4; a7 == a5; a7 == a6; a7 == a7; a7 == a8;\n a8 == a1; a8 == a2; a8 == a3; a8 == a4; a8 == a5; a8 == a6; a8 == a7; a8 == a8;\n\n a1 == a1; a1 == a2; a1 == a3; a1 == a4; a1 == a5; a1 == a6; a1 == a7; a1 == a8;\n a2 == a1; a2 == a2; a2 == a3; a2 == a4; a2 == a5; a2 == a6; a2 == a7; a2 == a8;\n a3 == a1; a3 == a2; a3 == a3; a3 == a4; a3 == a5; a3 == a6; a3 == a7; a3 == a8;\n a4 == a1; a4 == a2; a4 == a3; a4 == a4; a4 == a5; a4 == a6; a4 == a7; a4 == a8;\n a5 == a1; a5 == a2; a5 == a3; a5 == a4; a5 == a5; a5 == a6; a5 == a7; a5 == a8;\n a6 == a1; a6 == a2; a6 == a3; a6 == a4; a6 == a5; a6 == a6; a6 == a7; a6 == a8;\n a7 == a1; a7 == a2; a7 == a3; a7 == a4; a7 == a5; a7 == a6; a7 == a7; a7 == a8;\n a8 == a1; a8 == a2; a8 == a3; a8 == a4; a8 == a5; a8 == a6; a8 == a7; a8 == a8;\n\n}\n\nvar elapsed = clock() - start;\nprint \"loop\";\nprint loopTime;\nprint \"elapsed\";\nprint elapsed;\nprint \"equals\";\nprint elapsed - loopTime;\n" }, { "path": "test/benchmark/trees.lox", "content": "class Tree {\n init(depth) {\n this.depth = depth;\n if (depth > 0) {\n this.a = Tree(depth - 1);\n this.b = Tree(depth - 1);\n this.c = Tree(depth - 1);\n this.d = Tree(depth - 1);\n this.e = Tree(depth - 1);\n }\n }\n\n walk() {\n if (this.depth == 0) return 0;\n return this.depth \n + this.a.walk()\n + this.b.walk()\n + this.c.walk()\n + this.d.walk()\n + this.e.walk();\n }\n}\n\nvar tree = Tree(8);\nvar start = clock();\nfor (var i = 0; i < 100; i = i + 1) {\n if (tree.walk() != 122068) print \"Error\";\n}\nprint clock() - start;\n" }, { "path": "test/benchmark/zoo.lox", "content": "class Zoo {\n init() {\n this.aarvark = 1;\n this.baboon = 1;\n this.cat = 1;\n this.donkey = 1;\n this.elephant = 1;\n this.fox = 1;\n }\n ant() { return this.aarvark; }\n banana() { return this.baboon; }\n tuna() { return this.cat; }\n hay() { return this.donkey; }\n grass() { return this.elephant; }\n mouse() { return this.fox; }\n}\n\nvar zoo = Zoo();\nvar sum = 0;\nvar start = clock();\nwhile (sum < 10000000) {\n sum = sum + zoo.ant()\n + zoo.banana()\n + zoo.tuna()\n + zoo.hay()\n + zoo.grass()\n + zoo.mouse();\n}\n\nprint sum;\nprint clock() - start;\n" }, { "path": "test/benchmark/zoo_batch.lox", "content": "class Zoo {\n init() {\n this.aarvark = 1;\n this.baboon = 1;\n this.cat = 1;\n this.donkey = 1;\n this.elephant = 1;\n this.fox = 1;\n }\n ant() { return this.aarvark; }\n banana() { return this.baboon; }\n tuna() { return this.cat; }\n hay() { return this.donkey; }\n grass() { return this.elephant; }\n mouse() { return this.fox; }\n}\n\nvar zoo = Zoo();\nvar sum = 0;\nvar start = clock();\nvar batch = 0;\nwhile (clock() - start < 10) {\n for (var i = 0; i < 10000; i = i + 1) {\n sum = sum + zoo.ant()\n + zoo.banana()\n + zoo.tuna()\n + zoo.hay()\n + zoo.grass()\n + zoo.mouse();\n }\n batch = batch + 1;\n}\n\nprint sum;\nprint batch;\nprint clock() - start;\n" }, { "path": "test/block/empty.lox", "content": "{} // By itself.\n\n// In a statement.\nif (true) {}\nif (false) {} else {}\n\nprint \"ok\"; // expect: ok\n" }, { "path": "test/block/scope.lox", "content": "var a = \"outer\";\n\n{\n var a = \"inner\";\n print a; // expect: inner\n}\n\nprint a; // expect: outer\n" }, { "path": "test/bool/equality.lox", "content": "print true == true; // expect: true\nprint true == false; // expect: false\nprint false == true; // expect: false\nprint false == false; // expect: true\n\n// Not equal to other types.\nprint true == 1; // expect: false\nprint false == 0; // expect: false\nprint true == \"true\"; // expect: false\nprint false == \"false\"; // expect: false\nprint false == \"\"; // expect: false\n\nprint true != true; // expect: false\nprint true != false; // expect: true\nprint false != true; // expect: true\nprint false != false; // expect: false\n\n// Not equal to other types.\nprint true != 1; // expect: true\nprint false != 0; // expect: true\nprint true != \"true\"; // expect: true\nprint false != \"false\"; // expect: true\nprint false != \"\"; // expect: true\n" }, { "path": "test/bool/not.lox", "content": "print !true; // expect: false\nprint !false; // expect: true\nprint !!true; // expect: true\n" }, { "path": "test/call/bool.lox", "content": "true(); // expect runtime error: Can only call functions and classes.\n" }, { "path": "test/call/nil.lox", "content": "nil(); // expect runtime error: Can only call functions and classes.\n" }, { "path": "test/call/num.lox", "content": "123(); // expect runtime error: Can only call functions and classes.\n" }, { "path": "test/call/object.lox", "content": "class Foo {}\n\nvar foo = Foo();\nfoo(); // expect runtime error: Can only call functions and classes.\n" }, { "path": "test/call/string.lox", "content": "\"str\"(); // expect runtime error: Can only call functions and classes.\n" }, { "path": "test/class/empty.lox", "content": "class Foo {}\n\nprint Foo; // expect: Foo\n" }, { "path": "test/class/inherit_self.lox", "content": "class Foo < Foo {} // Error at 'Foo': A class can't inherit from itself.\n" }, { "path": "test/class/inherited_method.lox", "content": "class Foo {\n inFoo() {\n print \"in foo\";\n }\n}\n\nclass Bar < Foo {\n inBar() {\n print \"in bar\";\n }\n}\n\nclass Baz < Bar {\n inBaz() {\n print \"in baz\";\n }\n}\n\nvar baz = Baz();\nbaz.inFoo(); // expect: in foo\nbaz.inBar(); // expect: in bar\nbaz.inBaz(); // expect: in baz\n" }, { "path": "test/class/local_inherit_other.lox", "content": "class A {}\n\nfun f() {\n class B < A {}\n return B;\n}\n\nprint f(); // expect: B\n" }, { "path": "test/class/local_inherit_self.lox", "content": "{\n class Foo < Foo {} // Error at 'Foo': A class can't inherit from itself.\n}\n// [c line 5] Error at end: Expect '}' after block.\n" }, { "path": "test/class/local_reference_self.lox", "content": "{\n class Foo {\n returnSelf() {\n return Foo;\n }\n }\n\n print Foo().returnSelf(); // expect: Foo\n}\n" }, { "path": "test/class/reference_self.lox", "content": "class Foo {\n returnSelf() {\n return Foo;\n }\n}\n\nprint Foo().returnSelf(); // expect: Foo\n" }, { "path": "test/closure/assign_to_closure.lox", "content": "var f;\nvar g;\n\n{\n var local = \"local\";\n fun f_() {\n print local;\n local = \"after f\";\n print local;\n }\n f = f_;\n\n fun g_() {\n print local;\n local = \"after g\";\n print local;\n }\n g = g_;\n}\n\nf();\n// expect: local\n// expect: after f\n\ng();\n// expect: after f\n// expect: after g\n" }, { "path": "test/closure/assign_to_shadowed_later.lox", "content": "var a = \"global\";\n\n{\n fun assign() {\n a = \"assigned\";\n }\n\n var a = \"inner\";\n assign();\n print a; // expect: inner\n}\n\nprint a; // expect: assigned\n" }, { "path": "test/closure/close_over_function_parameter.lox", "content": "var f;\n\nfun foo(param) {\n fun f_() {\n print param;\n }\n f = f_;\n}\nfoo(\"param\");\n\nf(); // expect: param\n" }, { "path": "test/closure/close_over_later_variable.lox", "content": "// This is a regression test. There was a bug where if an upvalue for an\n// earlier local (here \"a\") was captured *after* a later one (\"b\"), then it\n// would crash because it walked to the end of the upvalue list (correct), but\n// then didn't handle not finding the variable.\n\nfun f() {\n var a = \"a\";\n var b = \"b\";\n fun g() {\n print b; // expect: b\n print a; // expect: a\n }\n g();\n}\nf();\n" }, { "path": "test/closure/close_over_method_parameter.lox", "content": "var f;\n\nclass Foo {\n method(param) {\n fun f_() {\n print param;\n }\n f = f_;\n }\n}\n\nFoo().method(\"param\");\nf(); // expect: param\n" }, { "path": "test/closure/closed_closure_in_function.lox", "content": "var f;\n\n{\n var local = \"local\";\n fun f_() {\n print local;\n }\n f = f_;\n}\n\nf(); // expect: local\n" }, { "path": "test/closure/nested_closure.lox", "content": "var f;\n\nfun f1() {\n var a = \"a\";\n fun f2() {\n var b = \"b\";\n fun f3() {\n var c = \"c\";\n fun f4() {\n print a;\n print b;\n print c;\n }\n f = f4;\n }\n f3();\n }\n f2();\n}\nf1();\n\nf();\n// expect: a\n// expect: b\n// expect: c\n" }, { "path": "test/closure/open_closure_in_function.lox", "content": "{\n var local = \"local\";\n fun f() {\n print local; // expect: local\n }\n f();\n}\n" }, { "path": "test/closure/reference_closure_multiple_times.lox", "content": "var f;\n\n{\n var a = \"a\";\n fun f_() {\n print a;\n print a;\n }\n f = f_;\n}\n\nf();\n// expect: a\n// expect: a\n" }, { "path": "test/closure/reuse_closure_slot.lox", "content": "{\n var f;\n\n {\n var a = \"a\";\n fun f_() { print a; }\n f = f_;\n }\n\n {\n // Since a is out of scope, the local slot will be reused by b. Make sure\n // that f still closes over a.\n var b = \"b\";\n f(); // expect: a\n }\n}\n" }, { "path": "test/closure/shadow_closure_with_local.lox", "content": "{\n var foo = \"closure\";\n fun f() {\n {\n print foo; // expect: closure\n var foo = \"shadow\";\n print foo; // expect: shadow\n }\n print foo; // expect: closure\n }\n f();\n}\n" }, { "path": "test/closure/unused_closure.lox", "content": "// This is a regression test. There was a bug where the VM would try to close\n// an upvalue even if the upvalue was never created because the codepath for\n// the closure was not executed.\n\n{\n var a = \"a\";\n if (false) {\n fun foo() { a; }\n }\n}\n\n// If we get here, we didn't segfault when a went out of scope.\nprint \"ok\"; // expect: ok\n" }, { "path": "test/closure/unused_later_closure.lox", "content": "// This is a regression test. When closing upvalues for discarded locals, it\n// wouldn't make sure it discarded the upvalue for the correct stack slot.\n//\n// Here we create two locals that can be closed over, but only the first one\n// actually is. When \"b\" goes out of scope, we need to make sure we don't\n// prematurely close \"a\".\nvar closure;\n\n{\n var a = \"a\";\n\n {\n var b = \"b\";\n fun returnA() {\n return a;\n }\n\n closure = returnA;\n\n if (false) {\n fun returnB() {\n return b;\n }\n }\n }\n\n print closure(); // expect: a\n}\n" }, { "path": "test/comments/line_at_eof.lox", "content": "print \"ok\"; // expect: ok\n// comment" }, { "path": "test/comments/only_line_comment.lox", "content": "// comment" }, { "path": "test/comments/only_line_comment_and_line.lox", "content": "// comment\n" }, { "path": "test/comments/unicode.lox", "content": "// Unicode characters are allowed in comments.\n//\n// Latin 1 Supplement: £§¶ÜÞ\n// Latin Extended-A: ĐĦŋœ\n// Latin Extended-B: ƂƢƩǁ\n// Other stuff: ឃᢆ᯽₪ℜ↩⊗┺░\n// Emoji: ☃☺♣\n\nprint \"ok\"; // expect: ok\n" }, { "path": "test/constructor/arguments.lox", "content": "class Foo {\n init(a, b) {\n print \"init\"; // expect: init\n this.a = a;\n this.b = b;\n }\n}\n\nvar foo = Foo(1, 2);\nprint foo.a; // expect: 1\nprint foo.b; // expect: 2\n" }, { "path": "test/constructor/call_init_early_return.lox", "content": "class Foo {\n init() {\n print \"init\";\n return;\n print \"nope\";\n }\n}\n\nvar foo = Foo(); // expect: init\nprint foo.init(); // expect: init\n// expect: Foo instance\n" }, { "path": "test/constructor/call_init_explicitly.lox", "content": "class Foo {\n init(arg) {\n print \"Foo.init(\" + arg + \")\";\n this.field = \"init\";\n }\n}\n\nvar foo = Foo(\"one\"); // expect: Foo.init(one)\nfoo.field = \"field\";\n\nvar foo2 = foo.init(\"two\"); // expect: Foo.init(two)\nprint foo2; // expect: Foo instance\n\n// Make sure init() doesn't create a fresh instance.\nprint foo.field; // expect: init\n" }, { "path": "test/constructor/default.lox", "content": "class Foo {}\n\nvar foo = Foo();\nprint foo; // expect: Foo instance\n" }, { "path": "test/constructor/default_arguments.lox", "content": "class Foo {}\n\nvar foo = Foo(1, 2, 3); // expect runtime error: Expected 0 arguments but got 3.\n" }, { "path": "test/constructor/early_return.lox", "content": "class Foo {\n init() {\n print \"init\";\n return;\n print \"nope\";\n }\n}\n\nvar foo = Foo(); // expect: init\nprint foo; // expect: Foo instance\n" }, { "path": "test/constructor/extra_arguments.lox", "content": "class Foo {\n init(a, b) {\n this.a = a;\n this.b = b;\n }\n}\n\nvar foo = Foo(1, 2, 3, 4); // expect runtime error: Expected 2 arguments but got 4." }, { "path": "test/constructor/init_not_method.lox", "content": "class Foo {\n init(arg) {\n print \"Foo.init(\" + arg + \")\";\n this.field = \"init\";\n }\n}\n\nfun init() {\n print \"not initializer\";\n}\n\ninit(); // expect: not initializer\n" }, { "path": "test/constructor/missing_arguments.lox", "content": "class Foo {\n init(a, b) {}\n}\n\nvar foo = Foo(1); // expect runtime error: Expected 2 arguments but got 1.\n" }, { "path": "test/constructor/return_in_nested_function.lox", "content": "class Foo {\n init() {\n fun init() {\n return \"bar\";\n }\n print init(); // expect: bar\n }\n}\n\nprint Foo(); // expect: Foo instance\n" }, { "path": "test/constructor/return_value.lox", "content": "class Foo {\n init() {\n return \"result\"; // Error at 'return': Can't return a value from an initializer.\n }\n}\n" }, { "path": "test/empty_file.lox", "content": "" }, { "path": "test/expressions/evaluate.lox", "content": "// Note: This is just for the expression evaluating chapter which evaluates an\n// expression directly.\n(5 - (3 - 1)) + -1\n// expect: 2\n" }, { "path": "test/expressions/parse.lox", "content": "// Note: This is just for the expression parsing chapter which prints the AST.\n(5 - (3 - 1)) + -1\n// expect: (+ (group (- 5.0 (group (- 3.0 1.0)))) (- 1.0))\n" }, { "path": "test/field/call_function_field.lox", "content": "class Foo {}\n\nfun bar(a, b) {\n print \"bar\";\n print a;\n print b;\n}\n\nvar foo = Foo();\nfoo.bar = bar;\n\nfoo.bar(1, 2);\n// expect: bar\n// expect: 1\n// expect: 2\n" }, { "path": "test/field/call_nonfunction_field.lox", "content": "class Foo {}\n\nvar foo = Foo();\nfoo.bar = \"not fn\";\n\nfoo.bar(); // expect runtime error: Can only call functions and classes.\n" }, { "path": "test/field/get_and_set_method.lox", "content": "// Bound methods have identity equality.\nclass Foo {\n method(a) {\n print \"method\";\n print a;\n }\n other(a) {\n print \"other\";\n print a;\n }\n}\n\nvar foo = Foo();\nvar method = foo.method;\n\n// Setting a property shadows the instance method.\nfoo.method = foo.other;\nfoo.method(1);\n// expect: other\n// expect: 1\n\n// The old method handle still points to the original method.\nmethod(2);\n// expect: method\n// expect: 2\n" }, { "path": "test/field/get_on_bool.lox", "content": "true.foo; // expect runtime error: Only instances have properties.\n" }, { "path": "test/field/get_on_class.lox", "content": "class Foo {}\nFoo.bar; // expect runtime error: Only instances have properties.\n" }, { "path": "test/field/get_on_function.lox", "content": "fun foo() {}\n\nfoo.bar; // expect runtime error: Only instances have properties.\n" }, { "path": "test/field/get_on_nil.lox", "content": "nil.foo; // expect runtime error: Only instances have properties.\n" }, { "path": "test/field/get_on_num.lox", "content": "123.foo; // expect runtime error: Only instances have properties.\n" }, { "path": "test/field/get_on_string.lox", "content": "\"str\".foo; // expect runtime error: Only instances have properties.\n" }, { "path": "test/field/many.lox", "content": "class Foo {}\n\nvar foo = Foo();\nfun setFields() {\n foo.bilberry = \"bilberry\";\n foo.lime = \"lime\";\n foo.elderberry = \"elderberry\";\n foo.raspberry = \"raspberry\";\n foo.gooseberry = \"gooseberry\";\n foo.longan = \"longan\";\n foo.mandarine = \"mandarine\";\n foo.kiwifruit = \"kiwifruit\";\n foo.orange = \"orange\";\n foo.pomegranate = \"pomegranate\";\n foo.tomato = \"tomato\";\n foo.banana = \"banana\";\n foo.juniper = \"juniper\";\n foo.damson = \"damson\";\n foo.blackcurrant = \"blackcurrant\";\n foo.peach = \"peach\";\n foo.grape = \"grape\";\n foo.mango = \"mango\";\n foo.redcurrant = \"redcurrant\";\n foo.watermelon = \"watermelon\";\n foo.plumcot = \"plumcot\";\n foo.papaya = \"papaya\";\n foo.cloudberry = \"cloudberry\";\n foo.rambutan = \"rambutan\";\n foo.salak = \"salak\";\n foo.physalis = \"physalis\";\n foo.huckleberry = \"huckleberry\";\n foo.coconut = \"coconut\";\n foo.date = \"date\";\n foo.tamarind = \"tamarind\";\n foo.lychee = \"lychee\";\n foo.raisin = \"raisin\";\n foo.apple = \"apple\";\n foo.avocado = \"avocado\";\n foo.nectarine = \"nectarine\";\n foo.pomelo = \"pomelo\";\n foo.melon = \"melon\";\n foo.currant = \"currant\";\n foo.plum = \"plum\";\n foo.persimmon = \"persimmon\";\n foo.olive = \"olive\";\n foo.cranberry = \"cranberry\";\n foo.boysenberry = \"boysenberry\";\n foo.blackberry = \"blackberry\";\n foo.passionfruit = \"passionfruit\";\n foo.mulberry = \"mulberry\";\n foo.marionberry = \"marionberry\";\n foo.plantain = \"plantain\";\n foo.lemon = \"lemon\";\n foo.yuzu = \"yuzu\";\n foo.loquat = \"loquat\";\n foo.kumquat = \"kumquat\";\n foo.salmonberry = \"salmonberry\";\n foo.tangerine = \"tangerine\";\n foo.durian = \"durian\";\n foo.pear = \"pear\";\n foo.cantaloupe = \"cantaloupe\";\n foo.quince = \"quince\";\n foo.guava = \"guava\";\n foo.strawberry = \"strawberry\";\n foo.nance = \"nance\";\n foo.apricot = \"apricot\";\n foo.jambul = \"jambul\";\n foo.grapefruit = \"grapefruit\";\n foo.clementine = \"clementine\";\n foo.jujube = \"jujube\";\n foo.cherry = \"cherry\";\n foo.feijoa = \"feijoa\";\n foo.jackfruit = \"jackfruit\";\n foo.fig = \"fig\";\n foo.cherimoya = \"cherimoya\";\n foo.pineapple = \"pineapple\";\n foo.blueberry = \"blueberry\";\n foo.jabuticaba = \"jabuticaba\";\n foo.miracle = \"miracle\";\n foo.dragonfruit = \"dragonfruit\";\n foo.satsuma = \"satsuma\";\n foo.tamarillo = \"tamarillo\";\n foo.honeydew = \"honeydew\";\n}\n\nsetFields();\n\nfun printFields() {\n print foo.apple; // expect: apple\n print foo.apricot; // expect: apricot\n print foo.avocado; // expect: avocado\n print foo.banana; // expect: banana\n print foo.bilberry; // expect: bilberry\n print foo.blackberry; // expect: blackberry\n print foo.blackcurrant; // expect: blackcurrant\n print foo.blueberry; // expect: blueberry\n print foo.boysenberry; // expect: boysenberry\n print foo.cantaloupe; // expect: cantaloupe\n print foo.cherimoya; // expect: cherimoya\n print foo.cherry; // expect: cherry\n print foo.clementine; // expect: clementine\n print foo.cloudberry; // expect: cloudberry\n print foo.coconut; // expect: coconut\n print foo.cranberry; // expect: cranberry\n print foo.currant; // expect: currant\n print foo.damson; // expect: damson\n print foo.date; // expect: date\n print foo.dragonfruit; // expect: dragonfruit\n print foo.durian; // expect: durian\n print foo.elderberry; // expect: elderberry\n print foo.feijoa; // expect: feijoa\n print foo.fig; // expect: fig\n print foo.gooseberry; // expect: gooseberry\n print foo.grape; // expect: grape\n print foo.grapefruit; // expect: grapefruit\n print foo.guava; // expect: guava\n print foo.honeydew; // expect: honeydew\n print foo.huckleberry; // expect: huckleberry\n print foo.jabuticaba; // expect: jabuticaba\n print foo.jackfruit; // expect: jackfruit\n print foo.jambul; // expect: jambul\n print foo.jujube; // expect: jujube\n print foo.juniper; // expect: juniper\n print foo.kiwifruit; // expect: kiwifruit\n print foo.kumquat; // expect: kumquat\n print foo.lemon; // expect: lemon\n print foo.lime; // expect: lime\n print foo.longan; // expect: longan\n print foo.loquat; // expect: loquat\n print foo.lychee; // expect: lychee\n print foo.mandarine; // expect: mandarine\n print foo.mango; // expect: mango\n print foo.marionberry; // expect: marionberry\n print foo.melon; // expect: melon\n print foo.miracle; // expect: miracle\n print foo.mulberry; // expect: mulberry\n print foo.nance; // expect: nance\n print foo.nectarine; // expect: nectarine\n print foo.olive; // expect: olive\n print foo.orange; // expect: orange\n print foo.papaya; // expect: papaya\n print foo.passionfruit; // expect: passionfruit\n print foo.peach; // expect: peach\n print foo.pear; // expect: pear\n print foo.persimmon; // expect: persimmon\n print foo.physalis; // expect: physalis\n print foo.pineapple; // expect: pineapple\n print foo.plantain; // expect: plantain\n print foo.plum; // expect: plum\n print foo.plumcot; // expect: plumcot\n print foo.pomegranate; // expect: pomegranate\n print foo.pomelo; // expect: pomelo\n print foo.quince; // expect: quince\n print foo.raisin; // expect: raisin\n print foo.rambutan; // expect: rambutan\n print foo.raspberry; // expect: raspberry\n print foo.redcurrant; // expect: redcurrant\n print foo.salak; // expect: salak\n print foo.salmonberry; // expect: salmonberry\n print foo.satsuma; // expect: satsuma\n print foo.strawberry; // expect: strawberry\n print foo.tamarillo; // expect: tamarillo\n print foo.tamarind; // expect: tamarind\n print foo.tangerine; // expect: tangerine\n print foo.tomato; // expect: tomato\n print foo.watermelon; // expect: watermelon\n print foo.yuzu; // expect: yuzu\n}\n\nprintFields();\n" }, { "path": "test/field/method.lox", "content": "class Foo {\n bar(arg) {\n print arg;\n }\n}\n\nvar bar = Foo().bar;\nprint \"got method\"; // expect: got method\nbar(\"arg\"); // expect: arg\n" }, { "path": "test/field/method_binds_this.lox", "content": "class Foo {\n sayName(a) {\n print this.name;\n print a;\n }\n}\n\nvar foo1 = Foo();\nfoo1.name = \"foo1\";\n\nvar foo2 = Foo();\nfoo2.name = \"foo2\";\n\n// Store the method reference on another object.\nfoo2.fn = foo1.sayName;\n// Still retains original receiver.\nfoo2.fn(1);\n// expect: foo1\n// expect: 1\n" }, { "path": "test/field/on_instance.lox", "content": "class Foo {}\n\nvar foo = Foo();\n\nprint foo.bar = \"bar value\"; // expect: bar value\nprint foo.baz = \"baz value\"; // expect: baz value\n\nprint foo.bar; // expect: bar value\nprint foo.baz; // expect: baz value\n" }, { "path": "test/field/set_evaluation_order.lox", "content": "undefined1.bar // expect runtime error: Undefined variable 'undefined1'.\n = undefined2;" }, { "path": "test/field/set_on_bool.lox", "content": "true.foo = \"value\"; // expect runtime error: Only instances have fields.\n" }, { "path": "test/field/set_on_class.lox", "content": "class Foo {}\nFoo.bar = \"value\"; // expect runtime error: Only instances have fields.\n" }, { "path": "test/field/set_on_function.lox", "content": "fun foo() {}\n\nfoo.bar = \"value\"; // expect runtime error: Only instances have fields.\n" }, { "path": "test/field/set_on_nil.lox", "content": "nil.foo = \"value\"; // expect runtime error: Only instances have fields.\n" }, { "path": "test/field/set_on_num.lox", "content": "123.foo = \"value\"; // expect runtime error: Only instances have fields.\n" }, { "path": "test/field/set_on_string.lox", "content": "\"str\".foo = \"value\"; // expect runtime error: Only instances have fields.\n" }, { "path": "test/field/undefined.lox", "content": "class Foo {}\nvar foo = Foo();\n\nfoo.bar; // expect runtime error: Undefined property 'bar'.\n" }, { "path": "test/for/class_in_body.lox", "content": "// [line 2] Error at 'class': Expect expression.\nfor (;;) class Foo {}\n" }, { "path": "test/for/closure_in_body.lox", "content": "var f1;\nvar f2;\nvar f3;\n\nfor (var i = 1; i < 4; i = i + 1) {\n var j = i;\n fun f() {\n print i;\n print j;\n }\n\n if (j == 1) f1 = f;\n else if (j == 2) f2 = f;\n else f3 = f;\n}\n\nf1(); // expect: 4\n // expect: 1\nf2(); // expect: 4\n // expect: 2\nf3(); // expect: 4\n // expect: 3\n" }, { "path": "test/for/fun_in_body.lox", "content": "// [line 2] Error at 'fun': Expect expression.\nfor (;;) fun foo() {}\n" }, { "path": "test/for/return_closure.lox", "content": "fun f() {\n for (;;) {\n var i = \"i\";\n fun g() { print i; }\n return g;\n }\n}\n\nvar h = f();\nh(); // expect: i\n" }, { "path": "test/for/return_inside.lox", "content": "fun f() {\n for (;;) {\n var i = \"i\";\n return i;\n }\n}\n\nprint f();\n// expect: i\n" }, { "path": "test/for/scope.lox", "content": "{\n var i = \"before\";\n\n // New variable is in inner scope.\n for (var i = 0; i < 1; i = i + 1) {\n print i; // expect: 0\n\n // Loop body is in second inner scope.\n var i = -1;\n print i; // expect: -1\n }\n}\n\n{\n // New variable shadows outer variable.\n for (var i = 0; i > 0; i = i + 1) {}\n\n // Goes out of scope after loop.\n var i = \"after\";\n print i; // expect: after\n\n // Can reuse an existing variable.\n for (i = 0; i < 1; i = i + 1) {\n print i; // expect: 0\n }\n}\n" }, { "path": "test/for/statement_condition.lox", "content": "// [line 3] Error at '{': Expect expression.\n// [line 3] Error at ')': Expect ';' after expression.\nfor (var a = 1; {}; a = a + 1) {}\n" }, { "path": "test/for/statement_increment.lox", "content": "// [line 2] Error at '{': Expect expression.\nfor (var a = 1; a < 2; {}) {}\n" }, { "path": "test/for/statement_initializer.lox", "content": "// [line 3] Error at '{': Expect expression.\n// [line 3] Error at ')': Expect ';' after expression.\nfor ({}; a < 2; a = a + 1) {}\n" }, { "path": "test/for/syntax.lox", "content": "// Single-expression body.\nfor (var c = 0; c < 3;) print c = c + 1;\n// expect: 1\n// expect: 2\n// expect: 3\n\n// Block body.\nfor (var a = 0; a < 3; a = a + 1) {\n print a;\n}\n// expect: 0\n// expect: 1\n// expect: 2\n\n// No clauses.\nfun foo() {\n for (;;) return \"done\";\n}\nprint foo(); // expect: done\n\n// No variable.\nvar i = 0;\nfor (; i < 2; i = i + 1) print i;\n// expect: 0\n// expect: 1\n\n// No condition.\nfun bar() {\n for (var i = 0;; i = i + 1) {\n print i;\n if (i >= 2) return;\n }\n}\nbar();\n// expect: 0\n// expect: 1\n// expect: 2\n\n// No increment.\nfor (var i = 0; i < 2;) {\n print i;\n i = i + 1;\n}\n// expect: 0\n// expect: 1\n\n// Statement bodies.\nfor (; false;) if (true) 1; else 2;\nfor (; false;) while (true) 1;\nfor (; false;) for (;;) 1;\n" }, { "path": "test/for/var_in_body.lox", "content": "// [line 2] Error at 'var': Expect expression.\nfor (;;) var foo;\n" }, { "path": "test/function/body_must_be_block.lox", "content": "// [line 3] Error at '123': Expect '{' before function body.\n// [c line 4] Error at end: Expect '}' after block.\nfun f() 123;\n" }, { "path": "test/function/empty_body.lox", "content": "fun f() {}\nprint f(); // expect: nil\n" }, { "path": "test/function/extra_arguments.lox", "content": "fun f(a, b) {\n print a;\n print b;\n}\n\nf(1, 2, 3, 4); // expect runtime error: Expected 2 arguments but got 4.\n" }, { "path": "test/function/local_mutual_recursion.lox", "content": "{\n fun isEven(n) {\n if (n == 0) return true;\n return isOdd(n - 1); // expect runtime error: Undefined variable 'isOdd'.\n }\n\n fun isOdd(n) {\n if (n == 0) return false;\n return isEven(n - 1);\n }\n\n isEven(4);\n}" }, { "path": "test/function/local_recursion.lox", "content": "{\n fun fib(n) {\n if (n < 2) return n;\n return fib(n - 1) + fib(n - 2);\n }\n\n print fib(8); // expect: 21\n}\n" }, { "path": "test/function/missing_arguments.lox", "content": "fun f(a, b) {}\n\nf(1); // expect runtime error: Expected 2 arguments but got 1.\n" }, { "path": "test/function/missing_comma_in_parameters.lox", "content": "// [line 3] Error at 'c': Expect ')' after parameters.\n// [c line 4] Error at end: Expect '}' after block.\nfun foo(a, b c, d, e, f) {}\n" }, { "path": "test/function/mutual_recursion.lox", "content": "fun isEven(n) {\n if (n == 0) return true;\n return isOdd(n - 1);\n}\n\nfun isOdd(n) {\n if (n == 0) return false;\n return isEven(n - 1);\n}\n\nprint isEven(4); // expect: true\nprint isOdd(3); // expect: true\n" }, { "path": "test/function/nested_call_with_arguments.lox", "content": "fun returnArg(arg) {\n return arg;\n}\n\nfun returnFunCallWithArg(func, arg) {\n return returnArg(func)(arg);\n}\n\nfun printArg(arg) {\n print arg;\n}\n\nreturnFunCallWithArg(printArg, \"hello world\"); // expect: hello world\n" }, { "path": "test/function/parameters.lox", "content": "fun f0() { return 0; }\nprint f0(); // expect: 0\n\nfun f1(a) { return a; }\nprint f1(1); // expect: 1\n\nfun f2(a, b) { return a + b; }\nprint f2(1, 2); // expect: 3\n\nfun f3(a, b, c) { return a + b + c; }\nprint f3(1, 2, 3); // expect: 6\n\nfun f4(a, b, c, d) { return a + b + c + d; }\nprint f4(1, 2, 3, 4); // expect: 10\n\nfun f5(a, b, c, d, e) { return a + b + c + d + e; }\nprint f5(1, 2, 3, 4, 5); // expect: 15\n\nfun f6(a, b, c, d, e, f) { return a + b + c + d + e + f; }\nprint f6(1, 2, 3, 4, 5, 6); // expect: 21\n\nfun f7(a, b, c, d, e, f, g) { return a + b + c + d + e + f + g; }\nprint f7(1, 2, 3, 4, 5, 6, 7); // expect: 28\n\nfun f8(a, b, c, d, e, f, g, h) { return a + b + c + d + e + f + g + h; }\nprint f8(1, 2, 3, 4, 5, 6, 7, 8); // expect: 36\n" }, { "path": "test/function/print.lox", "content": "fun foo() {}\nprint foo; // expect: \n\nprint clock; // expect: \n" }, { "path": "test/function/recursion.lox", "content": "fun fib(n) {\n if (n < 2) return n;\n return fib(n - 1) + fib(n - 2);\n}\n\nprint fib(8); // expect: 21\n" }, { "path": "test/function/too_many_arguments.lox", "content": "fun foo() {}\n{\n var a = 1;\n foo(\n a, // 1\n a, // 2\n a, // 3\n a, // 4\n a, // 5\n a, // 6\n a, // 7\n a, // 8\n a, // 9\n a, // 10\n a, // 11\n a, // 12\n a, // 13\n a, // 14\n a, // 15\n a, // 16\n a, // 17\n a, // 18\n a, // 19\n a, // 20\n a, // 21\n a, // 22\n a, // 23\n a, // 24\n a, // 25\n a, // 26\n a, // 27\n a, // 28\n a, // 29\n a, // 30\n a, // 31\n a, // 32\n a, // 33\n a, // 34\n a, // 35\n a, // 36\n a, // 37\n a, // 38\n a, // 39\n a, // 40\n a, // 41\n a, // 42\n a, // 43\n a, // 44\n a, // 45\n a, // 46\n a, // 47\n a, // 48\n a, // 49\n a, // 50\n a, // 51\n a, // 52\n a, // 53\n a, // 54\n a, // 55\n a, // 56\n a, // 57\n a, // 58\n a, // 59\n a, // 60\n a, // 61\n a, // 62\n a, // 63\n a, // 64\n a, // 65\n a, // 66\n a, // 67\n a, // 68\n a, // 69\n a, // 70\n a, // 71\n a, // 72\n a, // 73\n a, // 74\n a, // 75\n a, // 76\n a, // 77\n a, // 78\n a, // 79\n a, // 80\n a, // 81\n a, // 82\n a, // 83\n a, // 84\n a, // 85\n a, // 86\n a, // 87\n a, // 88\n a, // 89\n a, // 90\n a, // 91\n a, // 92\n a, // 93\n a, // 94\n a, // 95\n a, // 96\n a, // 97\n a, // 98\n a, // 99\n a, // 100\n a, // 101\n a, // 102\n a, // 103\n a, // 104\n a, // 105\n a, // 106\n a, // 107\n a, // 108\n a, // 109\n a, // 110\n a, // 111\n a, // 112\n a, // 113\n a, // 114\n a, // 115\n a, // 116\n a, // 117\n a, // 118\n a, // 119\n a, // 120\n a, // 121\n a, // 122\n a, // 123\n a, // 124\n a, // 125\n a, // 126\n a, // 127\n a, // 128\n a, // 129\n a, // 130\n a, // 131\n a, // 132\n a, // 133\n a, // 134\n a, // 135\n a, // 136\n a, // 137\n a, // 138\n a, // 139\n a, // 140\n a, // 141\n a, // 142\n a, // 143\n a, // 144\n a, // 145\n a, // 146\n a, // 147\n a, // 148\n a, // 149\n a, // 150\n a, // 151\n a, // 152\n a, // 153\n a, // 154\n a, // 155\n a, // 156\n a, // 157\n a, // 158\n a, // 159\n a, // 160\n a, // 161\n a, // 162\n a, // 163\n a, // 164\n a, // 165\n a, // 166\n a, // 167\n a, // 168\n a, // 169\n a, // 170\n a, // 171\n a, // 172\n a, // 173\n a, // 174\n a, // 175\n a, // 176\n a, // 177\n a, // 178\n a, // 179\n a, // 180\n a, // 181\n a, // 182\n a, // 183\n a, // 184\n a, // 185\n a, // 186\n a, // 187\n a, // 188\n a, // 189\n a, // 190\n a, // 191\n a, // 192\n a, // 193\n a, // 194\n a, // 195\n a, // 196\n a, // 197\n a, // 198\n a, // 199\n a, // 200\n a, // 201\n a, // 202\n a, // 203\n a, // 204\n a, // 205\n a, // 206\n a, // 207\n a, // 208\n a, // 209\n a, // 210\n a, // 211\n a, // 212\n a, // 213\n a, // 214\n a, // 215\n a, // 216\n a, // 217\n a, // 218\n a, // 219\n a, // 220\n a, // 221\n a, // 222\n a, // 223\n a, // 224\n a, // 225\n a, // 226\n a, // 227\n a, // 228\n a, // 229\n a, // 230\n a, // 231\n a, // 232\n a, // 233\n a, // 234\n a, // 235\n a, // 236\n a, // 237\n a, // 238\n a, // 239\n a, // 240\n a, // 241\n a, // 242\n a, // 243\n a, // 244\n a, // 245\n a, // 246\n a, // 247\n a, // 248\n a, // 249\n a, // 250\n a, // 251\n a, // 252\n a, // 253\n a, // 254\n a, // 255\n a); // Error at 'a': Can't have more than 255 arguments.\n}\n" }, { "path": "test/function/too_many_parameters.lox", "content": "// 256 parameters.\nfun f(\n a1,\n a2,\n a3,\n a4,\n a5,\n a6,\n a7,\n a8,\n a9,\n a10,\n a11,\n a12,\n a13,\n a14,\n a15,\n a16,\n a17,\n a18,\n a19,\n a20,\n a21,\n a22,\n a23,\n a24,\n a25,\n a26,\n a27,\n a28,\n a29,\n a30,\n a31,\n a32,\n a33,\n a34,\n a35,\n a36,\n a37,\n a38,\n a39,\n a40,\n a41,\n a42,\n a43,\n a44,\n a45,\n a46,\n a47,\n a48,\n a49,\n a50,\n a51,\n a52,\n a53,\n a54,\n a55,\n a56,\n a57,\n a58,\n a59,\n a60,\n a61,\n a62,\n a63,\n a64,\n a65,\n a66,\n a67,\n a68,\n a69,\n a70,\n a71,\n a72,\n a73,\n a74,\n a75,\n a76,\n a77,\n a78,\n a79,\n a80,\n a81,\n a82,\n a83,\n a84,\n a85,\n a86,\n a87,\n a88,\n a89,\n a90,\n a91,\n a92,\n a93,\n a94,\n a95,\n a96,\n a97,\n a98,\n a99,\n a100,\n a101,\n a102,\n a103,\n a104,\n a105,\n a106,\n a107,\n a108,\n a109,\n a110,\n a111,\n a112,\n a113,\n a114,\n a115,\n a116,\n a117,\n a118,\n a119,\n a120,\n a121,\n a122,\n a123,\n a124,\n a125,\n a126,\n a127,\n a128,\n a129,\n a130,\n a131,\n a132,\n a133,\n a134,\n a135,\n a136,\n a137,\n a138,\n a139,\n a140,\n a141,\n a142,\n a143,\n a144,\n a145,\n a146,\n a147,\n a148,\n a149,\n a150,\n a151,\n a152,\n a153,\n a154,\n a155,\n a156,\n a157,\n a158,\n a159,\n a160,\n a161,\n a162,\n a163,\n a164,\n a165,\n a166,\n a167,\n a168,\n a169,\n a170,\n a171,\n a172,\n a173,\n a174,\n a175,\n a176,\n a177,\n a178,\n a179,\n a180,\n a181,\n a182,\n a183,\n a184,\n a185,\n a186,\n a187,\n a188,\n a189,\n a190,\n a191,\n a192,\n a193,\n a194,\n a195,\n a196,\n a197,\n a198,\n a199,\n a200,\n a201,\n a202,\n a203,\n a204,\n a205,\n a206,\n a207,\n a208,\n a209,\n a210,\n a211,\n a212,\n a213,\n a214,\n a215,\n a216,\n a217,\n a218,\n a219,\n a220,\n a221,\n a222,\n a223,\n a224,\n a225,\n a226,\n a227,\n a228,\n a229,\n a230,\n a231,\n a232,\n a233,\n a234,\n a235,\n a236,\n a237,\n a238,\n a239,\n a240,\n a241,\n a242,\n a243,\n a244,\n a245,\n a246,\n a247,\n a248,\n a249,\n a250,\n a251,\n a252,\n a253,\n a254,\n a255, a) {} // Error at 'a': Can't have more than 255 parameters.\n" }, { "path": "test/if/class_in_else.lox", "content": "// [line 2] Error at 'class': Expect expression.\nif (true) \"ok\"; else class Foo {}\n" }, { "path": "test/if/class_in_then.lox", "content": "// [line 2] Error at 'class': Expect expression.\nif (true) class Foo {}\n" }, { "path": "test/if/dangling_else.lox", "content": "// A dangling else binds to the right-most if.\nif (true) if (false) print \"bad\"; else print \"good\"; // expect: good\nif (false) if (true) print \"bad\"; else print \"bad\";\n" }, { "path": "test/if/else.lox", "content": "// Evaluate the 'else' expression if the condition is false.\nif (true) print \"good\"; else print \"bad\"; // expect: good\nif (false) print \"bad\"; else print \"good\"; // expect: good\n\n// Allow block body.\nif (false) nil; else { print \"block\"; } // expect: block\n" }, { "path": "test/if/fun_in_else.lox", "content": "// [line 2] Error at 'fun': Expect expression.\nif (true) \"ok\"; else fun foo() {}\n" }, { "path": "test/if/fun_in_then.lox", "content": "// [line 2] Error at 'fun': Expect expression.\nif (true) fun foo() {}\n" }, { "path": "test/if/if.lox", "content": "// Evaluate the 'then' expression if the condition is true.\nif (true) print \"good\"; // expect: good\nif (false) print \"bad\";\n\n// Allow block body.\nif (true) { print \"block\"; } // expect: block\n\n// Assignment in if condition.\nvar a = false;\nif (a = true) print a; // expect: true\n" }, { "path": "test/if/truth.lox", "content": "// False and nil are false.\nif (false) print \"bad\"; else print \"false\"; // expect: false\nif (nil) print \"bad\"; else print \"nil\"; // expect: nil\n\n// Everything else is true.\nif (true) print true; // expect: true\nif (0) print 0; // expect: 0\nif (\"\") print \"empty\"; // expect: empty\n" }, { "path": "test/if/var_in_else.lox", "content": "// [line 2] Error at 'var': Expect expression.\nif (true) \"ok\"; else var foo;\n" }, { "path": "test/if/var_in_then.lox", "content": "// [line 2] Error at 'var': Expect expression.\nif (true) var foo;\n" }, { "path": "test/inheritance/constructor.lox", "content": "class A {\n init(param) {\n this.field = param;\n }\n\n test() {\n print this.field;\n }\n}\n\nclass B < A {}\n\nvar b = B(\"value\");\nb.test(); // expect: value\n" }, { "path": "test/inheritance/inherit_from_function.lox", "content": "fun foo() {}\n\nclass Subclass < foo {} // expect runtime error: Superclass must be a class.\n" }, { "path": "test/inheritance/inherit_from_nil.lox", "content": "var Nil = nil;\nclass Foo < Nil {} // expect runtime error: Superclass must be a class.\n" }, { "path": "test/inheritance/inherit_from_number.lox", "content": "var Number = 123;\nclass Foo < Number {} // expect runtime error: Superclass must be a class.\n" }, { "path": "test/inheritance/inherit_methods.lox", "content": "class Foo {\n methodOnFoo() { print \"foo\"; }\n override() { print \"foo\"; }\n}\n\nclass Bar < Foo {\n methodOnBar() { print \"bar\"; }\n override() { print \"bar\"; }\n}\n\nvar bar = Bar();\nbar.methodOnFoo(); // expect: foo\nbar.methodOnBar(); // expect: bar\nbar.override(); // expect: bar\n" }, { "path": "test/inheritance/parenthesized_superclass.lox", "content": "class Foo {}\n\n// [line 4] Error at '(': Expect superclass name.\nclass Bar < (Foo) {}\n" }, { "path": "test/inheritance/set_fields_from_base_class.lox", "content": "class Foo {\n foo(a, b) {\n this.field1 = a;\n this.field2 = b;\n }\n\n fooPrint() {\n print this.field1;\n print this.field2;\n }\n}\n\nclass Bar < Foo {\n bar(a, b) {\n this.field1 = a;\n this.field2 = b;\n }\n\n barPrint() {\n print this.field1;\n print this.field2;\n }\n}\n\nvar bar = Bar();\nbar.foo(\"foo 1\", \"foo 2\");\nbar.fooPrint();\n// expect: foo 1\n// expect: foo 2\n\nbar.bar(\"bar 1\", \"bar 2\");\nbar.barPrint();\n// expect: bar 1\n// expect: bar 2\n\nbar.fooPrint();\n// expect: bar 1\n// expect: bar 2\n" }, { "path": "test/limit/loop_too_large.lox", "content": "var a = 0;\nwhile (false) {\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil; nil;\n} // Error at '}': Loop body too large.\n" }, { "path": "test/limit/no_reuse_constants.lox", "content": "fun f() {\n 0; 1; 2; 3; 4; 5; 6; 7;\n 8; 9; 10; 11; 12; 13; 14; 15;\n 16; 17; 18; 19; 20; 21; 22; 23;\n 24; 25; 26; 27; 28; 29; 30; 31;\n 32; 33; 34; 35; 36; 37; 38; 39;\n 40; 41; 42; 43; 44; 45; 46; 47;\n 48; 49; 50; 51; 52; 53; 54; 55;\n 56; 57; 58; 59; 60; 61; 62; 63;\n 64; 65; 66; 67; 68; 69; 70; 71;\n 72; 73; 74; 75; 76; 77; 78; 79;\n 80; 81; 82; 83; 84; 85; 86; 87;\n 88; 89; 90; 91; 92; 93; 94; 95;\n 96; 97; 98; 99; 100; 101; 102; 103;\n 104; 105; 106; 107; 108; 109; 110; 111;\n 112; 113; 114; 115; 116; 117; 118; 119;\n 120; 121; 122; 123; 124; 125; 126; 127;\n 128; 129; 130; 131; 132; 133; 134; 135;\n 136; 137; 138; 139; 140; 141; 142; 143;\n 144; 145; 146; 147; 148; 149; 150; 151;\n 152; 153; 154; 155; 156; 157; 158; 159;\n 160; 161; 162; 163; 164; 165; 166; 167;\n 168; 169; 170; 171; 172; 173; 174; 175;\n 176; 177; 178; 179; 180; 181; 182; 183;\n 184; 185; 186; 187; 188; 189; 190; 191;\n 192; 193; 194; 195; 196; 197; 198; 199;\n 200; 201; 202; 203; 204; 205; 206; 207;\n 208; 209; 210; 211; 212; 213; 214; 215;\n 216; 217; 218; 219; 220; 221; 222; 223;\n 224; 225; 226; 227; 228; 229; 230; 231;\n 232; 233; 234; 235; 236; 237; 238; 239;\n 240; 241; 242; 243; 244; 245; 246; 247;\n 248; 249; 250; 251; 252; 253; 254; 255;\n\n 1; // Error at '1': Too many constants in one chunk.\n}\n" }, { "path": "test/limit/stack_overflow.lox", "content": "fun foo() {\n var a1;\n var a2;\n var a3;\n var a4;\n var a5;\n var a6;\n var a7;\n var a8;\n var a9;\n var a10;\n var a11;\n var a12;\n var a13;\n var a14;\n var a15;\n var a16;\n foo(); // expect runtime error: Stack overflow.\n}\n\nfoo();\n" }, { "path": "test/limit/too_many_constants.lox", "content": "fun f() {\n 0; 1; 2; 3; 4; 5; 6; 7;\n 8; 9; 10; 11; 12; 13; 14; 15;\n 16; 17; 18; 19; 20; 21; 22; 23;\n 24; 25; 26; 27; 28; 29; 30; 31;\n 32; 33; 34; 35; 36; 37; 38; 39;\n 40; 41; 42; 43; 44; 45; 46; 47;\n 48; 49; 50; 51; 52; 53; 54; 55;\n 56; 57; 58; 59; 60; 61; 62; 63;\n 64; 65; 66; 67; 68; 69; 70; 71;\n 72; 73; 74; 75; 76; 77; 78; 79;\n 80; 81; 82; 83; 84; 85; 86; 87;\n 88; 89; 90; 91; 92; 93; 94; 95;\n 96; 97; 98; 99; 100; 101; 102; 103;\n 104; 105; 106; 107; 108; 109; 110; 111;\n 112; 113; 114; 115; 116; 117; 118; 119;\n 120; 121; 122; 123; 124; 125; 126; 127;\n 128; 129; 130; 131; 132; 133; 134; 135;\n 136; 137; 138; 139; 140; 141; 142; 143;\n 144; 145; 146; 147; 148; 149; 150; 151;\n 152; 153; 154; 155; 156; 157; 158; 159;\n 160; 161; 162; 163; 164; 165; 166; 167;\n 168; 169; 170; 171; 172; 173; 174; 175;\n 176; 177; 178; 179; 180; 181; 182; 183;\n 184; 185; 186; 187; 188; 189; 190; 191;\n 192; 193; 194; 195; 196; 197; 198; 199;\n 200; 201; 202; 203; 204; 205; 206; 207;\n 208; 209; 210; 211; 212; 213; 214; 215;\n 216; 217; 218; 219; 220; 221; 222; 223;\n 224; 225; 226; 227; 228; 229; 230; 231;\n 232; 233; 234; 235; 236; 237; 238; 239;\n 240; 241; 242; 243; 244; 245; 246; 247;\n 248; 249; 250; 251; 252; 253; 254; 255;\n\n \"oops\"; // Error at '\"oops\"': Too many constants in one chunk.\n}\n" }, { "path": "test/limit/too_many_locals.lox", "content": "fun f() {\n // var v00; First slot already taken.\n\n var v01; var v02; var v03; var v04; var v05; var v06; var v07;\n var v08; var v09; var v0a; var v0b; var v0c; var v0d; var v0e; var v0f;\n\n var v10; var v11; var v12; var v13; var v14; var v15; var v16; var v17;\n var v18; var v19; var v1a; var v1b; var v1c; var v1d; var v1e; var v1f;\n\n var v20; var v21; var v22; var v23; var v24; var v25; var v26; var v27;\n var v28; var v29; var v2a; var v2b; var v2c; var v2d; var v2e; var v2f;\n\n var v30; var v31; var v32; var v33; var v34; var v35; var v36; var v37;\n var v38; var v39; var v3a; var v3b; var v3c; var v3d; var v3e; var v3f;\n\n var v40; var v41; var v42; var v43; var v44; var v45; var v46; var v47;\n var v48; var v49; var v4a; var v4b; var v4c; var v4d; var v4e; var v4f;\n\n var v50; var v51; var v52; var v53; var v54; var v55; var v56; var v57;\n var v58; var v59; var v5a; var v5b; var v5c; var v5d; var v5e; var v5f;\n\n var v60; var v61; var v62; var v63; var v64; var v65; var v66; var v67;\n var v68; var v69; var v6a; var v6b; var v6c; var v6d; var v6e; var v6f;\n\n var v70; var v71; var v72; var v73; var v74; var v75; var v76; var v77;\n var v78; var v79; var v7a; var v7b; var v7c; var v7d; var v7e; var v7f;\n\n var v80; var v81; var v82; var v83; var v84; var v85; var v86; var v87;\n var v88; var v89; var v8a; var v8b; var v8c; var v8d; var v8e; var v8f;\n\n var v90; var v91; var v92; var v93; var v94; var v95; var v96; var v97;\n var v98; var v99; var v9a; var v9b; var v9c; var v9d; var v9e; var v9f;\n\n var va0; var va1; var va2; var va3; var va4; var va5; var va6; var va7;\n var va8; var va9; var vaa; var vab; var vac; var vad; var vae; var vaf;\n\n var vb0; var vb1; var vb2; var vb3; var vb4; var vb5; var vb6; var vb7;\n var vb8; var vb9; var vba; var vbb; var vbc; var vbd; var vbe; var vbf;\n\n var vc0; var vc1; var vc2; var vc3; var vc4; var vc5; var vc6; var vc7;\n var vc8; var vc9; var vca; var vcb; var vcc; var vcd; var vce; var vcf;\n\n var vd0; var vd1; var vd2; var vd3; var vd4; var vd5; var vd6; var vd7;\n var vd8; var vd9; var vda; var vdb; var vdc; var vdd; var vde; var vdf;\n\n var ve0; var ve1; var ve2; var ve3; var ve4; var ve5; var ve6; var ve7;\n var ve8; var ve9; var vea; var veb; var vec; var ved; var vee; var vef;\n\n var vf0; var vf1; var vf2; var vf3; var vf4; var vf5; var vf6; var vf7;\n var vf8; var vf9; var vfa; var vfb; var vfc; var vfd; var vfe; var vff;\n\n var oops; // Error at 'oops': Too many local variables in function.\n}\n" }, { "path": "test/limit/too_many_upvalues.lox", "content": "fun f() {\n var v00; var v01; var v02; var v03; var v04; var v05; var v06; var v07;\n var v08; var v09; var v0a; var v0b; var v0c; var v0d; var v0e; var v0f;\n\n var v10; var v11; var v12; var v13; var v14; var v15; var v16; var v17;\n var v18; var v19; var v1a; var v1b; var v1c; var v1d; var v1e; var v1f;\n\n var v20; var v21; var v22; var v23; var v24; var v25; var v26; var v27;\n var v28; var v29; var v2a; var v2b; var v2c; var v2d; var v2e; var v2f;\n\n var v30; var v31; var v32; var v33; var v34; var v35; var v36; var v37;\n var v38; var v39; var v3a; var v3b; var v3c; var v3d; var v3e; var v3f;\n\n var v40; var v41; var v42; var v43; var v44; var v45; var v46; var v47;\n var v48; var v49; var v4a; var v4b; var v4c; var v4d; var v4e; var v4f;\n\n var v50; var v51; var v52; var v53; var v54; var v55; var v56; var v57;\n var v58; var v59; var v5a; var v5b; var v5c; var v5d; var v5e; var v5f;\n\n var v60; var v61; var v62; var v63; var v64; var v65; var v66; var v67;\n var v68; var v69; var v6a; var v6b; var v6c; var v6d; var v6e; var v6f;\n\n var v70; var v71; var v72; var v73; var v74; var v75; var v76; var v77;\n var v78; var v79; var v7a; var v7b; var v7c; var v7d; var v7e; var v7f;\n\n fun g() {\n var v80; var v81; var v82; var v83; var v84; var v85; var v86; var v87;\n var v88; var v89; var v8a; var v8b; var v8c; var v8d; var v8e; var v8f;\n\n var v90; var v91; var v92; var v93; var v94; var v95; var v96; var v97;\n var v98; var v99; var v9a; var v9b; var v9c; var v9d; var v9e; var v9f;\n\n var va0; var va1; var va2; var va3; var va4; var va5; var va6; var va7;\n var va8; var va9; var vaa; var vab; var vac; var vad; var vae; var vaf;\n\n var vb0; var vb1; var vb2; var vb3; var vb4; var vb5; var vb6; var vb7;\n var vb8; var vb9; var vba; var vbb; var vbc; var vbd; var vbe; var vbf;\n\n var vc0; var vc1; var vc2; var vc3; var vc4; var vc5; var vc6; var vc7;\n var vc8; var vc9; var vca; var vcb; var vcc; var vcd; var vce; var vcf;\n\n var vd0; var vd1; var vd2; var vd3; var vd4; var vd5; var vd6; var vd7;\n var vd8; var vd9; var vda; var vdb; var vdc; var vdd; var vde; var vdf;\n\n var ve0; var ve1; var ve2; var ve3; var ve4; var ve5; var ve6; var ve7;\n var ve8; var ve9; var vea; var veb; var vec; var ved; var vee; var vef;\n\n var vf0; var vf1; var vf2; var vf3; var vf4; var vf5; var vf6; var vf7;\n var vf8; var vf9; var vfa; var vfb; var vfc; var vfd; var vfe; var vff;\n\n var oops;\n\n fun h() {\n v00; v01; v02; v03; v04; v05; v06; v07;\n v08; v09; v0a; v0b; v0c; v0d; v0e; v0f;\n\n v10; v11; v12; v13; v14; v15; v16; v17;\n v18; v19; v1a; v1b; v1c; v1d; v1e; v1f;\n\n v20; v21; v22; v23; v24; v25; v26; v27;\n v28; v29; v2a; v2b; v2c; v2d; v2e; v2f;\n\n v30; v31; v32; v33; v34; v35; v36; v37;\n v38; v39; v3a; v3b; v3c; v3d; v3e; v3f;\n\n v40; v41; v42; v43; v44; v45; v46; v47;\n v48; v49; v4a; v4b; v4c; v4d; v4e; v4f;\n\n v50; v51; v52; v53; v54; v55; v56; v57;\n v58; v59; v5a; v5b; v5c; v5d; v5e; v5f;\n\n v60; v61; v62; v63; v64; v65; v66; v67;\n v68; v69; v6a; v6b; v6c; v6d; v6e; v6f;\n\n v70; v71; v72; v73; v74; v75; v76; v77;\n v78; v79; v7a; v7b; v7c; v7d; v7e; v7f;\n\n v80; v81; v82; v83; v84; v85; v86; v87;\n v88; v89; v8a; v8b; v8c; v8d; v8e; v8f;\n\n v90; v91; v92; v93; v94; v95; v96; v97;\n v98; v99; v9a; v9b; v9c; v9d; v9e; v9f;\n\n va0; va1; va2; va3; va4; va5; va6; va7;\n va8; va9; vaa; vab; vac; vad; vae; vaf;\n\n vb0; vb1; vb2; vb3; vb4; vb5; vb6; vb7;\n vb8; vb9; vba; vbb; vbc; vbd; vbe; vbf;\n\n vc0; vc1; vc2; vc3; vc4; vc5; vc6; vc7;\n vc8; vc9; vca; vcb; vcc; vcd; vce; vcf;\n\n vd0; vd1; vd2; vd3; vd4; vd5; vd6; vd7;\n vd8; vd9; vda; vdb; vdc; vdd; vde; vdf;\n\n ve0; ve1; ve2; ve3; ve4; ve5; ve6; ve7;\n ve8; ve9; vea; veb; vec; ved; vee; vef;\n\n vf0; vf1; vf2; vf3; vf4; vf5; vf6; vf7;\n vf8; vf9; vfa; vfb; vfc; vfd; vfe; vff;\n\n oops; // Error at 'oops': Too many closure variables in function.\n }\n }\n}\n" }, { "path": "test/logical_operator/and.lox", "content": "// Note: These tests implicitly depend on ints being truthy.\n\n// Return the first non-true argument.\nprint false and 1; // expect: false\nprint true and 1; // expect: 1\nprint 1 and 2 and false; // expect: false\n\n// Return the last argument if all are true.\nprint 1 and true; // expect: true\nprint 1 and 2 and 3; // expect: 3\n\n// Short-circuit at the first false argument.\nvar a = \"before\";\nvar b = \"before\";\n(a = true) and\n (b = false) and\n (a = \"bad\");\nprint a; // expect: true\nprint b; // expect: false\n" }, { "path": "test/logical_operator/and_truth.lox", "content": "// False and nil are false.\nprint false and \"bad\"; // expect: false\nprint nil and \"bad\"; // expect: nil\n\n// Everything else is true.\nprint true and \"ok\"; // expect: ok\nprint 0 and \"ok\"; // expect: ok\nprint \"\" and \"ok\"; // expect: ok\n" }, { "path": "test/logical_operator/or.lox", "content": "// Note: These tests implicitly depend on ints being truthy.\n\n// Return the first true argument.\nprint 1 or true; // expect: 1\nprint false or 1; // expect: 1\nprint false or false or true; // expect: true\n\n// Return the last argument if all are false.\nprint false or false; // expect: false\nprint false or false or false; // expect: false\n\n// Short-circuit at the first true argument.\nvar a = \"before\";\nvar b = \"before\";\n(a = false) or\n (b = true) or\n (a = \"bad\");\nprint a; // expect: false\nprint b; // expect: true\n" }, { "path": "test/logical_operator/or_truth.lox", "content": "// False and nil are false.\nprint false or \"ok\"; // expect: ok\nprint nil or \"ok\"; // expect: ok\n\n// Everything else is true.\nprint true or \"ok\"; // expect: true\nprint 0 or \"ok\"; // expect: 0\nprint \"s\" or \"ok\"; // expect: s\n" }, { "path": "test/method/arity.lox", "content": "class Foo {\n method0() { return \"no args\"; }\n method1(a) { return a; }\n method2(a, b) { return a + b; }\n method3(a, b, c) { return a + b + c; }\n method4(a, b, c, d) { return a + b + c + d; }\n method5(a, b, c, d, e) { return a + b + c + d + e; }\n method6(a, b, c, d, e, f) { return a + b + c + d + e + f; }\n method7(a, b, c, d, e, f, g) { return a + b + c + d + e + f + g; }\n method8(a, b, c, d, e, f, g, h) { return a + b + c + d + e + f + g + h; }\n}\n\nvar foo = Foo();\nprint foo.method0(); // expect: no args\nprint foo.method1(1); // expect: 1\nprint foo.method2(1, 2); // expect: 3\nprint foo.method3(1, 2, 3); // expect: 6\nprint foo.method4(1, 2, 3, 4); // expect: 10\nprint foo.method5(1, 2, 3, 4, 5); // expect: 15\nprint foo.method6(1, 2, 3, 4, 5, 6); // expect: 21\nprint foo.method7(1, 2, 3, 4, 5, 6, 7); // expect: 28\nprint foo.method8(1, 2, 3, 4, 5, 6, 7, 8); // expect: 36\n" }, { "path": "test/method/empty_block.lox", "content": "class Foo {\n bar() {}\n}\n\nprint Foo().bar(); // expect: nil\n" }, { "path": "test/method/extra_arguments.lox", "content": "class Foo {\n method(a, b) {\n print a;\n print b;\n }\n}\n\nFoo().method(1, 2, 3, 4); // expect runtime error: Expected 2 arguments but got 4.\n" }, { "path": "test/method/missing_arguments.lox", "content": "class Foo {\n method(a, b) {}\n}\n\nFoo().method(1); // expect runtime error: Expected 2 arguments but got 1.\n" }, { "path": "test/method/not_found.lox", "content": "class Foo {}\n\nFoo().unknown(); // expect runtime error: Undefined property 'unknown'.\n" }, { "path": "test/method/print_bound_method.lox", "content": "class Foo {\n method() { }\n}\nvar foo = Foo();\nprint foo.method; // expect: \n" }, { "path": "test/method/refer_to_name.lox", "content": "class Foo {\n method() {\n print method; // expect runtime error: Undefined variable 'method'.\n }\n}\n\nFoo().method();\n" }, { "path": "test/method/too_many_arguments.lox", "content": "{\n var a = 1;\n true.method(\n a, // 1\n a, // 2\n a, // 3\n a, // 4\n a, // 5\n a, // 6\n a, // 7\n a, // 8\n a, // 9\n a, // 10\n a, // 11\n a, // 12\n a, // 13\n a, // 14\n a, // 15\n a, // 16\n a, // 17\n a, // 18\n a, // 19\n a, // 20\n a, // 21\n a, // 22\n a, // 23\n a, // 24\n a, // 25\n a, // 26\n a, // 27\n a, // 28\n a, // 29\n a, // 30\n a, // 31\n a, // 32\n a, // 33\n a, // 34\n a, // 35\n a, // 36\n a, // 37\n a, // 38\n a, // 39\n a, // 40\n a, // 41\n a, // 42\n a, // 43\n a, // 44\n a, // 45\n a, // 46\n a, // 47\n a, // 48\n a, // 49\n a, // 50\n a, // 51\n a, // 52\n a, // 53\n a, // 54\n a, // 55\n a, // 56\n a, // 57\n a, // 58\n a, // 59\n a, // 60\n a, // 61\n a, // 62\n a, // 63\n a, // 64\n a, // 65\n a, // 66\n a, // 67\n a, // 68\n a, // 69\n a, // 70\n a, // 71\n a, // 72\n a, // 73\n a, // 74\n a, // 75\n a, // 76\n a, // 77\n a, // 78\n a, // 79\n a, // 80\n a, // 81\n a, // 82\n a, // 83\n a, // 84\n a, // 85\n a, // 86\n a, // 87\n a, // 88\n a, // 89\n a, // 90\n a, // 91\n a, // 92\n a, // 93\n a, // 94\n a, // 95\n a, // 96\n a, // 97\n a, // 98\n a, // 99\n a, // 100\n a, // 101\n a, // 102\n a, // 103\n a, // 104\n a, // 105\n a, // 106\n a, // 107\n a, // 108\n a, // 109\n a, // 110\n a, // 111\n a, // 112\n a, // 113\n a, // 114\n a, // 115\n a, // 116\n a, // 117\n a, // 118\n a, // 119\n a, // 120\n a, // 121\n a, // 122\n a, // 123\n a, // 124\n a, // 125\n a, // 126\n a, // 127\n a, // 128\n a, // 129\n a, // 130\n a, // 131\n a, // 132\n a, // 133\n a, // 134\n a, // 135\n a, // 136\n a, // 137\n a, // 138\n a, // 139\n a, // 140\n a, // 141\n a, // 142\n a, // 143\n a, // 144\n a, // 145\n a, // 146\n a, // 147\n a, // 148\n a, // 149\n a, // 150\n a, // 151\n a, // 152\n a, // 153\n a, // 154\n a, // 155\n a, // 156\n a, // 157\n a, // 158\n a, // 159\n a, // 160\n a, // 161\n a, // 162\n a, // 163\n a, // 164\n a, // 165\n a, // 166\n a, // 167\n a, // 168\n a, // 169\n a, // 170\n a, // 171\n a, // 172\n a, // 173\n a, // 174\n a, // 175\n a, // 176\n a, // 177\n a, // 178\n a, // 179\n a, // 180\n a, // 181\n a, // 182\n a, // 183\n a, // 184\n a, // 185\n a, // 186\n a, // 187\n a, // 188\n a, // 189\n a, // 190\n a, // 191\n a, // 192\n a, // 193\n a, // 194\n a, // 195\n a, // 196\n a, // 197\n a, // 198\n a, // 199\n a, // 200\n a, // 201\n a, // 202\n a, // 203\n a, // 204\n a, // 205\n a, // 206\n a, // 207\n a, // 208\n a, // 209\n a, // 210\n a, // 211\n a, // 212\n a, // 213\n a, // 214\n a, // 215\n a, // 216\n a, // 217\n a, // 218\n a, // 219\n a, // 220\n a, // 221\n a, // 222\n a, // 223\n a, // 224\n a, // 225\n a, // 226\n a, // 227\n a, // 228\n a, // 229\n a, // 230\n a, // 231\n a, // 232\n a, // 233\n a, // 234\n a, // 235\n a, // 236\n a, // 237\n a, // 238\n a, // 239\n a, // 240\n a, // 241\n a, // 242\n a, // 243\n a, // 244\n a, // 245\n a, // 246\n a, // 247\n a, // 248\n a, // 249\n a, // 250\n a, // 251\n a, // 252\n a, // 253\n a, // 254\n a, // 255\n a); // Error at 'a': Can't have more than 255 arguments.\n}\n" }, { "path": "test/method/too_many_parameters.lox", "content": "class Foo {\n // 256 parameters.\n method(\n a1,\n a2,\n a3,\n a4,\n a5,\n a6,\n a7,\n a8,\n a9,\n a10,\n a11,\n a12,\n a13,\n a14,\n a15,\n a16,\n a17,\n a18,\n a19,\n a20,\n a21,\n a22,\n a23,\n a24,\n a25,\n a26,\n a27,\n a28,\n a29,\n a30,\n a31,\n a32,\n a33,\n a34,\n a35,\n a36,\n a37,\n a38,\n a39,\n a40,\n a41,\n a42,\n a43,\n a44,\n a45,\n a46,\n a47,\n a48,\n a49,\n a50,\n a51,\n a52,\n a53,\n a54,\n a55,\n a56,\n a57,\n a58,\n a59,\n a60,\n a61,\n a62,\n a63,\n a64,\n a65,\n a66,\n a67,\n a68,\n a69,\n a70,\n a71,\n a72,\n a73,\n a74,\n a75,\n a76,\n a77,\n a78,\n a79,\n a80,\n a81,\n a82,\n a83,\n a84,\n a85,\n a86,\n a87,\n a88,\n a89,\n a90,\n a91,\n a92,\n a93,\n a94,\n a95,\n a96,\n a97,\n a98,\n a99,\n a100,\n a101,\n a102,\n a103,\n a104,\n a105,\n a106,\n a107,\n a108,\n a109,\n a110,\n a111,\n a112,\n a113,\n a114,\n a115,\n a116,\n a117,\n a118,\n a119,\n a120,\n a121,\n a122,\n a123,\n a124,\n a125,\n a126,\n a127,\n a128,\n a129,\n a130,\n a131,\n a132,\n a133,\n a134,\n a135,\n a136,\n a137,\n a138,\n a139,\n a140,\n a141,\n a142,\n a143,\n a144,\n a145,\n a146,\n a147,\n a148,\n a149,\n a150,\n a151,\n a152,\n a153,\n a154,\n a155,\n a156,\n a157,\n a158,\n a159,\n a160,\n a161,\n a162,\n a163,\n a164,\n a165,\n a166,\n a167,\n a168,\n a169,\n a170,\n a171,\n a172,\n a173,\n a174,\n a175,\n a176,\n a177,\n a178,\n a179,\n a180,\n a181,\n a182,\n a183,\n a184,\n a185,\n a186,\n a187,\n a188,\n a189,\n a190,\n a191,\n a192,\n a193,\n a194,\n a195,\n a196,\n a197,\n a198,\n a199,\n a200,\n a201,\n a202,\n a203,\n a204,\n a205,\n a206,\n a207,\n a208,\n a209,\n a210,\n a211,\n a212,\n a213,\n a214,\n a215,\n a216,\n a217,\n a218,\n a219,\n a220,\n a221,\n a222,\n a223,\n a224,\n a225,\n a226,\n a227,\n a228,\n a229,\n a230,\n a231,\n a232,\n a233,\n a234,\n a235,\n a236,\n a237,\n a238,\n a239,\n a240,\n a241,\n a242,\n a243,\n a244,\n a245,\n a246,\n a247,\n a248,\n a249,\n a250,\n a251,\n a252,\n a253,\n a254,\n a255, a) {} // Error at 'a': Can't have more than 255 parameters.\n}\n" }, { "path": "test/nil/literal.lox", "content": "print nil; // expect: nil\n" }, { "path": "test/number/decimal_point_at_eof.lox", "content": "// [line 2] Error at end: Expect property name after '.'.\n123." }, { "path": "test/number/leading_dot.lox", "content": "// [line 2] Error at '.': Expect expression.\n.123;\n" }, { "path": "test/number/literals.lox", "content": "print 123; // expect: 123\nprint 987654; // expect: 987654\nprint 0; // expect: 0\nprint -0; // expect: -0\n\nprint 123.456; // expect: 123.456\nprint -0.001; // expect: -0.001\n" }, { "path": "test/number/nan_equality.lox", "content": "var nan = 0/0;\n\nprint nan == 0; // expect: false\nprint nan != 1; // expect: true\n\n// NaN is not equal to self.\nprint nan == nan; // expect: false\nprint nan != nan; // expect: true\n" }, { "path": "test/number/trailing_dot.lox", "content": "// [line 2] Error at ';': Expect property name after '.'.\n123.;\n" }, { "path": "test/operator/add.lox", "content": "print 123 + 456; // expect: 579\nprint \"str\" + \"ing\"; // expect: string\n" }, { "path": "test/operator/add_bool_nil.lox", "content": "true + nil; // expect runtime error: Operands must be two numbers or two strings.\n" }, { "path": "test/operator/add_bool_num.lox", "content": "true + 123; // expect runtime error: Operands must be two numbers or two strings.\n" }, { "path": "test/operator/add_bool_string.lox", "content": "true + \"s\"; // expect runtime error: Operands must be two numbers or two strings.\n" }, { "path": "test/operator/add_nil_nil.lox", "content": "nil + nil; // expect runtime error: Operands must be two numbers or two strings.\n" }, { "path": "test/operator/add_num_nil.lox", "content": "1 + nil; // expect runtime error: Operands must be two numbers or two strings.\n" }, { "path": "test/operator/add_string_nil.lox", "content": "\"s\" + nil; // expect runtime error: Operands must be two numbers or two strings.\n" }, { "path": "test/operator/comparison.lox", "content": "print 1 < 2; // expect: true\nprint 2 < 2; // expect: false\nprint 2 < 1; // expect: false\n\nprint 1 <= 2; // expect: true\nprint 2 <= 2; // expect: true\nprint 2 <= 1; // expect: false\n\nprint 1 > 2; // expect: false\nprint 2 > 2; // expect: false\nprint 2 > 1; // expect: true\n\nprint 1 >= 2; // expect: false\nprint 2 >= 2; // expect: true\nprint 2 >= 1; // expect: true\n\n// Zero and negative zero compare the same.\nprint 0 < -0; // expect: false\nprint -0 < 0; // expect: false\nprint 0 > -0; // expect: false\nprint -0 > 0; // expect: false\nprint 0 <= -0; // expect: true\nprint -0 <= 0; // expect: true\nprint 0 >= -0; // expect: true\nprint -0 >= 0; // expect: true\n" }, { "path": "test/operator/divide.lox", "content": "print 8 / 2; // expect: 4\nprint 12.34 / 12.34; // expect: 1\n" }, { "path": "test/operator/divide_nonnum_num.lox", "content": "\"1\" / 1; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/divide_num_nonnum.lox", "content": "1 / \"1\"; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/equals.lox", "content": "print nil == nil; // expect: true\n\nprint true == true; // expect: true\nprint true == false; // expect: false\n\nprint 1 == 1; // expect: true\nprint 1 == 2; // expect: false\n\nprint \"str\" == \"str\"; // expect: true\nprint \"str\" == \"ing\"; // expect: false\n\nprint nil == false; // expect: false\nprint false == 0; // expect: false\nprint 0 == \"0\"; // expect: false\n" }, { "path": "test/operator/equals_class.lox", "content": "// Bound methods have identity equality.\nclass Foo {}\nclass Bar {}\n\nprint Foo == Foo; // expect: true\nprint Foo == Bar; // expect: false\nprint Bar == Foo; // expect: false\nprint Bar == Bar; // expect: true\n\nprint Foo == \"Foo\"; // expect: false\nprint Foo == nil; // expect: false\nprint Foo == 123; // expect: false\nprint Foo == true; // expect: false\n" }, { "path": "test/operator/equals_method.lox", "content": "// Bound methods have identity equality.\nclass Foo {\n method() {}\n}\n\nvar foo = Foo();\nvar fooMethod = foo.method;\n\n// Same bound method.\nprint fooMethod == fooMethod; // expect: true\n\n// Different closurizations.\nprint foo.method == foo.method; // expect: false\n" }, { "path": "test/operator/greater_nonnum_num.lox", "content": "\"1\" > 1; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/greater_num_nonnum.lox", "content": "1 > \"1\"; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/greater_or_equal_nonnum_num.lox", "content": "\"1\" >= 1; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/greater_or_equal_num_nonnum.lox", "content": "1 >= \"1\"; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/less_nonnum_num.lox", "content": "\"1\" < 1; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/less_num_nonnum.lox", "content": "1 < \"1\"; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/less_or_equal_nonnum_num.lox", "content": "\"1\" <= 1; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/less_or_equal_num_nonnum.lox", "content": "1 <= \"1\"; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/multiply.lox", "content": "print 5 * 3; // expect: 15\nprint 12.34 * 0.3; // expect: 3.702\n" }, { "path": "test/operator/multiply_nonnum_num.lox", "content": "\"1\" * 1; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/multiply_num_nonnum.lox", "content": "1 * \"1\"; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/negate.lox", "content": "print -(3); // expect: -3\nprint --(3); // expect: 3\nprint ---(3); // expect: -3\n" }, { "path": "test/operator/negate_nonnum.lox", "content": "-\"s\"; // expect runtime error: Operand must be a number.\n" }, { "path": "test/operator/not.lox", "content": "print !true; // expect: false\nprint !false; // expect: true\nprint !!true; // expect: true\n\nprint !123; // expect: false\nprint !0; // expect: false\n\nprint !nil; // expect: true\n\nprint !\"\"; // expect: false\n\nfun foo() {}\nprint !foo; // expect: false\n" }, { "path": "test/operator/not_class.lox", "content": "class Bar {}\nprint !Bar; // expect: false\nprint !Bar(); // expect: false\n" }, { "path": "test/operator/not_equals.lox", "content": "print nil != nil; // expect: false\n\nprint true != true; // expect: false\nprint true != false; // expect: true\n\nprint 1 != 1; // expect: false\nprint 1 != 2; // expect: true\n\nprint \"str\" != \"str\"; // expect: false\nprint \"str\" != \"ing\"; // expect: true\n\nprint nil != false; // expect: true\nprint false != 0; // expect: true\nprint 0 != \"0\"; // expect: true\n" }, { "path": "test/operator/subtract.lox", "content": "print 4 - 3; // expect: 1\nprint 1.2 - 1.2; // expect: 0\n" }, { "path": "test/operator/subtract_nonnum_num.lox", "content": "\"1\" - 1; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/operator/subtract_num_nonnum.lox", "content": "1 - \"1\"; // expect runtime error: Operands must be numbers.\n" }, { "path": "test/precedence.lox", "content": "// * has higher precedence than +.\nprint 2 + 3 * 4; // expect: 14\n\n// * has higher precedence than -.\nprint 20 - 3 * 4; // expect: 8\n\n// / has higher precedence than +.\nprint 2 + 6 / 3; // expect: 4\n\n// / has higher precedence than -.\nprint 2 - 6 / 3; // expect: 0\n\n// < has higher precedence than ==.\nprint false == 2 < 1; // expect: true\n\n// > has higher precedence than ==.\nprint false == 1 > 2; // expect: true\n\n// <= has higher precedence than ==.\nprint false == 2 <= 1; // expect: true\n\n// >= has higher precedence than ==.\nprint false == 1 >= 2; // expect: true\n\n// 1 - 1 is not space-sensitive.\nprint 1 - 1; // expect: 0\nprint 1 -1; // expect: 0\nprint 1- 1; // expect: 0\nprint 1-1; // expect: 0\n\n// Using () for grouping.\nprint (2 * (6 - (2 + 2))); // expect: 4\n" }, { "path": "test/print/missing_argument.lox", "content": "// [line 2] Error at ';': Expect expression.\nprint;\n" }, { "path": "test/regression/394.lox", "content": "{\n class A {}\n class B < A {}\n print B; // expect: B\n}\n" }, { "path": "test/regression/40.lox", "content": "fun caller(g) {\n g();\n // g should be a function, not nil.\n print g == nil; // expect: false\n}\n\nfun callCaller() {\n var capturedVar = \"before\";\n var a = \"a\";\n\n fun f() {\n // Commenting the next line out prevents the bug!\n capturedVar = \"after\";\n\n // Returning anything also fixes it, even nil:\n //return nil;\n }\n\n caller(f);\n}\n\ncallCaller();\n" }, { "path": "test/return/after_else.lox", "content": "fun f() {\n if (false) \"no\"; else return \"ok\";\n}\n\nprint f(); // expect: ok\n" }, { "path": "test/return/after_if.lox", "content": "fun f() {\n if (true) return \"ok\";\n}\n\nprint f(); // expect: ok\n" }, { "path": "test/return/after_while.lox", "content": "fun f() {\n while (true) return \"ok\";\n}\n\nprint f(); // expect: ok\n" }, { "path": "test/return/at_top_level.lox", "content": "return \"wat\"; // Error at 'return': Can't return from top-level code.\n" }, { "path": "test/return/in_function.lox", "content": "fun f() {\n return \"ok\";\n print \"bad\";\n}\n\nprint f(); // expect: ok\n" }, { "path": "test/return/in_method.lox", "content": "class Foo {\n method() {\n return \"ok\";\n print \"bad\";\n }\n}\n\nprint Foo().method(); // expect: ok\n" }, { "path": "test/return/return_nil_if_no_value.lox", "content": "fun f() {\n return;\n print \"bad\";\n}\n\nprint f(); // expect: nil\n" }, { "path": "test/scanning/identifiers.lox", "content": "andy formless fo _ _123 _abc ab123\nabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890_\n\n// expect: IDENTIFIER andy null\n// expect: IDENTIFIER formless null\n// expect: IDENTIFIER fo null\n// expect: IDENTIFIER _ null\n// expect: IDENTIFIER _123 null\n// expect: IDENTIFIER _abc null\n// expect: IDENTIFIER ab123 null\n// expect: IDENTIFIER abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890_ null\n// expect: EOF null\n" }, { "path": "test/scanning/keywords.lox", "content": "and class else false for fun if nil or return super this true var while\n\n// expect: AND and null\n// expect: CLASS class null\n// expect: ELSE else null\n// expect: FALSE false null\n// expect: FOR for null\n// expect: FUN fun null\n// expect: IF if null\n// expect: NIL nil null\n// expect: OR or null\n// expect: RETURN return null\n// expect: SUPER super null\n// expect: THIS this null\n// expect: TRUE true null\n// expect: VAR var null\n// expect: WHILE while null\n// expect: EOF null\n" }, { "path": "test/scanning/numbers.lox", "content": "123\n123.456\n.456\n123.\n\n// expect: NUMBER 123 123.0\n// expect: NUMBER 123.456 123.456\n// expect: DOT . null\n// expect: NUMBER 456 456.0\n// expect: NUMBER 123 123.0\n// expect: DOT . null\n// expect: EOF null\n" }, { "path": "test/scanning/punctuators.lox", "content": "(){};,+-*!===<=>=!=<>/.\n\n// expect: LEFT_PAREN ( null\n// expect: RIGHT_PAREN ) null\n// expect: LEFT_BRACE { null\n// expect: RIGHT_BRACE } null\n// expect: SEMICOLON ; null\n// expect: COMMA , null\n// expect: PLUS + null\n// expect: MINUS - null\n// expect: STAR * null\n// expect: BANG_EQUAL != null\n// expect: EQUAL_EQUAL == null\n// expect: LESS_EQUAL <= null\n// expect: GREATER_EQUAL >= null\n// expect: BANG_EQUAL != null\n// expect: LESS < null\n// expect: GREATER > null\n// expect: SLASH / null\n// expect: DOT . null\n// expect: EOF null\n" }, { "path": "test/scanning/strings.lox", "content": "\"\"\n\"string\"\n\n// expect: STRING \"\" \n// expect: STRING \"string\" string\n// expect: EOF null" }, { "path": "test/scanning/whitespace.lox", "content": "space tabs\t\t\t\tnewlines\n\n\n\n\nend\n\n// expect: IDENTIFIER space null\n// expect: IDENTIFIER tabs null\n// expect: IDENTIFIER newlines null\n// expect: IDENTIFIER end null\n// expect: EOF null\n" }, { "path": "test/string/error_after_multiline.lox", "content": "// Tests that we correctly track the line info across multiline strings.\nvar a = \"1\n2\n3\n\";\n\nerr; // // expect runtime error: Undefined variable 'err'." }, { "path": "test/string/literals.lox", "content": "print \"(\" + \"\" + \")\"; // expect: ()\nprint \"a string\"; // expect: a string\n\n// Non-ASCII.\nprint \"A~¶Þॐஃ\"; // expect: A~¶Þॐஃ\n" }, { "path": "test/string/multiline.lox", "content": "var a = \"1\n2\n3\";\nprint a;\n// expect: 1\n// expect: 2\n// expect: 3\n" }, { "path": "test/string/unterminated.lox", "content": "// [line 2] Error: Unterminated string.\n\"this string has no close quote" }, { "path": "test/super/bound_method.lox", "content": "class A {\n method(arg) {\n print \"A.method(\" + arg + \")\";\n }\n}\n\nclass B < A {\n getClosure() {\n return super.method;\n }\n\n method(arg) {\n print \"B.method(\" + arg + \")\";\n }\n}\n\n\nvar closure = B().getClosure();\nclosure(\"arg\"); // expect: A.method(arg)\n" }, { "path": "test/super/call_other_method.lox", "content": "class Base {\n foo() {\n print \"Base.foo()\";\n }\n}\n\nclass Derived < Base {\n bar() {\n print \"Derived.bar()\";\n super.foo();\n }\n}\n\nDerived().bar();\n// expect: Derived.bar()\n// expect: Base.foo()\n" }, { "path": "test/super/call_same_method.lox", "content": "class Base {\n foo() {\n print \"Base.foo()\";\n }\n}\n\nclass Derived < Base {\n foo() {\n print \"Derived.foo()\";\n super.foo();\n }\n}\n\nDerived().foo();\n// expect: Derived.foo()\n// expect: Base.foo()\n" }, { "path": "test/super/closure.lox", "content": "class Base {\n toString() { return \"Base\"; }\n}\n\nclass Derived < Base {\n getClosure() {\n fun closure() {\n return super.toString();\n }\n return closure;\n }\n\n toString() { return \"Derived\"; }\n}\n\nvar closure = Derived().getClosure();\nprint closure(); // expect: Base\n" }, { "path": "test/super/constructor.lox", "content": "class Base {\n init(a, b) {\n print \"Base.init(\" + a + \", \" + b + \")\";\n }\n}\n\nclass Derived < Base {\n init() {\n print \"Derived.init()\";\n super.init(\"a\", \"b\");\n }\n}\n\nDerived();\n// expect: Derived.init()\n// expect: Base.init(a, b)\n" }, { "path": "test/super/extra_arguments.lox", "content": "class Base {\n foo(a, b) {\n print \"Base.foo(\" + a + \", \" + b + \")\";\n }\n}\n\nclass Derived < Base {\n foo() {\n print \"Derived.foo()\"; // expect: Derived.foo()\n super.foo(\"a\", \"b\", \"c\", \"d\"); // expect runtime error: Expected 2 arguments but got 4.\n }\n}\n\nDerived().foo();\n" }, { "path": "test/super/indirectly_inherited.lox", "content": "class A {\n foo() {\n print \"A.foo()\";\n }\n}\n\nclass B < A {}\n\nclass C < B {\n foo() {\n print \"C.foo()\";\n super.foo();\n }\n}\n\nC().foo();\n// expect: C.foo()\n// expect: A.foo()\n" }, { "path": "test/super/missing_arguments.lox", "content": "class Base {\n foo(a, b) {\n print \"Base.foo(\" + a + \", \" + b + \")\";\n }\n}\n\nclass Derived < Base {\n foo() {\n super.foo(1); // expect runtime error: Expected 2 arguments but got 1.\n }\n}\n\nDerived().foo();\n" }, { "path": "test/super/no_superclass_bind.lox", "content": "class Base {\n foo() {\n super.doesNotExist; // Error at 'super': Can't use 'super' in a class with no superclass.\n }\n}\n\nBase().foo();\n" }, { "path": "test/super/no_superclass_call.lox", "content": "class Base {\n foo() {\n super.doesNotExist(1); // Error at 'super': Can't use 'super' in a class with no superclass.\n }\n}\n\nBase().foo();\n" }, { "path": "test/super/no_superclass_method.lox", "content": "class Base {}\n\nclass Derived < Base {\n foo() {\n super.doesNotExist(1); // expect runtime error: Undefined property 'doesNotExist'.\n }\n}\n\nDerived().foo();\n" }, { "path": "test/super/parenthesized.lox", "content": "class A {\n method() {}\n}\n\nclass B < A {\n method() {\n // [line 8] Error at ')': Expect '.' after 'super'.\n (super).method();\n }\n}\n" }, { "path": "test/super/reassign_superclass.lox", "content": "class Base {\n method() {\n print \"Base.method()\";\n }\n}\n\nclass Derived < Base {\n method() {\n super.method();\n }\n}\n\nclass OtherBase {\n method() {\n print \"OtherBase.method()\";\n }\n}\n\nvar derived = Derived();\nderived.method(); // expect: Base.method()\nBase = OtherBase;\nderived.method(); // expect: Base.method()\n" }, { "path": "test/super/super_at_top_level.lox", "content": "super.foo(\"bar\"); // Error at 'super': Can't use 'super' outside of a class.\nsuper.foo; // Error at 'super': Can't use 'super' outside of a class." }, { "path": "test/super/super_in_closure_in_inherited_method.lox", "content": "class A {\n say() {\n print \"A\";\n }\n}\n\nclass B < A {\n getClosure() {\n fun closure() {\n super.say();\n }\n return closure;\n }\n\n say() {\n print \"B\";\n }\n}\n\nclass C < B {\n say() {\n print \"C\";\n }\n}\n\nC().getClosure()(); // expect: A\n" }, { "path": "test/super/super_in_inherited_method.lox", "content": "class A {\n say() {\n print \"A\";\n }\n}\n\nclass B < A {\n test() {\n super.say();\n }\n\n say() {\n print \"B\";\n }\n}\n\nclass C < B {\n say() {\n print \"C\";\n }\n}\n\nC().test(); // expect: A\n" }, { "path": "test/super/super_in_top_level_function.lox", "content": " super.bar(); // Error at 'super': Can't use 'super' outside of a class.\nfun foo() {\n}" }, { "path": "test/super/super_without_dot.lox", "content": "class A {}\n\nclass B < A {\n method() {\n // [line 6] Error at ';': Expect '.' after 'super'.\n super;\n }\n}\n" }, { "path": "test/super/super_without_name.lox", "content": "class A {}\n\nclass B < A {\n method() {\n super.; // Error at ';': Expect superclass method name.\n }\n}\n" }, { "path": "test/super/this_in_superclass_method.lox", "content": "class Base {\n init(a) {\n this.a = a;\n }\n}\n\nclass Derived < Base {\n init(a, b) {\n super.init(a);\n this.b = b;\n }\n}\n\nvar derived = Derived(\"a\", \"b\");\nprint derived.a; // expect: a\nprint derived.b; // expect: b\n" }, { "path": "test/this/closure.lox", "content": "class Foo {\n getClosure() {\n fun closure() {\n return this.toString();\n }\n return closure;\n }\n\n toString() { return \"Foo\"; }\n}\n\nvar closure = Foo().getClosure();\nprint closure(); // expect: Foo\n" }, { "path": "test/this/nested_class.lox", "content": "class Outer {\n method() {\n print this; // expect: Outer instance\n\n fun f() {\n print this; // expect: Outer instance\n\n class Inner {\n method() {\n print this; // expect: Inner instance\n }\n }\n\n Inner().method();\n }\n f();\n }\n}\n\nOuter().method();\n" }, { "path": "test/this/nested_closure.lox", "content": "class Foo {\n getClosure() {\n fun f() {\n fun g() {\n fun h() {\n return this.toString();\n }\n return h;\n }\n return g;\n }\n return f;\n }\n\n toString() { return \"Foo\"; }\n}\n\nvar closure = Foo().getClosure();\nprint closure()()(); // expect: Foo\n" }, { "path": "test/this/this_at_top_level.lox", "content": "this; // Error at 'this': Can't use 'this' outside of a class.\n" }, { "path": "test/this/this_in_method.lox", "content": "class Foo {\n bar() { return this; }\n baz() { return \"baz\"; }\n}\n\nprint Foo().bar().baz(); // expect: baz\n" }, { "path": "test/this/this_in_top_level_function.lox", "content": "fun foo() {\n this; // Error at 'this': Can't use 'this' outside of a class.\n}\n" }, { "path": "test/unexpected_character.lox", "content": "// [line 3] Error: Unexpected character.\n// [java line 3] Error at 'b': Expect ')' after arguments.\nfoo(a | b);\n" }, { "path": "test/variable/collide_with_parameter.lox", "content": "fun foo(a) {\n var a; // Error at 'a': Already a variable with this name in this scope.\n}\n" }, { "path": "test/variable/duplicate_local.lox", "content": "{\n var a = \"value\";\n var a = \"other\"; // Error at 'a': Already a variable with this name in this scope.\n}\n" }, { "path": "test/variable/duplicate_parameter.lox", "content": "fun foo(arg,\n arg) { // Error at 'arg': Already a variable with this name in this scope.\n \"body\";\n}\n" }, { "path": "test/variable/early_bound.lox", "content": "var a = \"outer\";\n{\n fun foo() {\n print a;\n }\n\n foo(); // expect: outer\n var a = \"inner\";\n foo(); // expect: outer\n}\n" }, { "path": "test/variable/in_middle_of_block.lox", "content": "{\n var a = \"a\";\n print a; // expect: a\n var b = a + \" b\";\n print b; // expect: a b\n var c = a + \" c\";\n print c; // expect: a c\n var d = b + \" d\";\n print d; // expect: a b d\n}\n" }, { "path": "test/variable/in_nested_block.lox", "content": "{\n var a = \"outer\";\n {\n print a; // expect: outer\n }\n}" }, { "path": "test/variable/local_from_method.lox", "content": "var foo = \"variable\";\n\nclass Foo {\n method() {\n print foo;\n }\n}\n\nFoo().method(); // expect: variable\n" }, { "path": "test/variable/redeclare_global.lox", "content": "var a = \"1\";\nvar a;\nprint a; // expect: nil\n" }, { "path": "test/variable/redefine_global.lox", "content": "var a = \"1\";\nvar a = \"2\";\nprint a; // expect: 2\n" }, { "path": "test/variable/scope_reuse_in_different_blocks.lox", "content": "{\n var a = \"first\";\n print a; // expect: first\n}\n\n{\n var a = \"second\";\n print a; // expect: second\n}\n" }, { "path": "test/variable/shadow_and_local.lox", "content": "{\n var a = \"outer\";\n {\n print a; // expect: outer\n var a = \"inner\";\n print a; // expect: inner\n }\n}" }, { "path": "test/variable/shadow_global.lox", "content": "var a = \"global\";\n{\n var a = \"shadow\";\n print a; // expect: shadow\n}\nprint a; // expect: global\n" }, { "path": "test/variable/shadow_local.lox", "content": "{\n var a = \"local\";\n {\n var a = \"shadow\";\n print a; // expect: shadow\n }\n print a; // expect: local\n}\n" }, { "path": "test/variable/undefined_global.lox", "content": "print notDefined; // expect runtime error: Undefined variable 'notDefined'.\n" }, { "path": "test/variable/undefined_local.lox", "content": "{\n print notDefined; // expect runtime error: Undefined variable 'notDefined'.\n}\n" }, { "path": "test/variable/uninitialized.lox", "content": "var a;\nprint a; // expect: nil\n" }, { "path": "test/variable/unreached_undefined.lox", "content": "if (false) {\n print notDefined;\n}\n\nprint \"ok\"; // expect: ok\n" }, { "path": "test/variable/use_false_as_var.lox", "content": "// [line 2] Error at 'false': Expect variable name.\nvar false = \"value\";\n" }, { "path": "test/variable/use_global_in_initializer.lox", "content": "var a = \"value\";\nvar a = a;\nprint a; // expect: value\n" }, { "path": "test/variable/use_local_in_initializer.lox", "content": "var a = \"outer\";\n{\n var a = a; // Error at 'a': Can't read local variable in its own initializer.\n}\n" }, { "path": "test/variable/use_nil_as_var.lox", "content": "// [line 2] Error at 'nil': Expect variable name.\nvar nil = \"value\";\n" }, { "path": "test/variable/use_this_as_var.lox", "content": "// [line 2] Error at 'this': Expect variable name.\nvar this = \"value\";\n" }, { "path": "test/while/class_in_body.lox", "content": "// [line 2] Error at 'class': Expect expression.\nwhile (true) class Foo {}\n" }, { "path": "test/while/closure_in_body.lox", "content": "var f1;\nvar f2;\nvar f3;\n\nvar i = 1;\nwhile (i < 4) {\n var j = i;\n fun f() { print j; }\n\n if (j == 1) f1 = f;\n else if (j == 2) f2 = f;\n else f3 = f;\n\n i = i + 1;\n}\n\nf1(); // expect: 1\nf2(); // expect: 2\nf3(); // expect: 3\n" }, { "path": "test/while/fun_in_body.lox", "content": "// [line 2] Error at 'fun': Expect expression.\nwhile (true) fun foo() {}\n" }, { "path": "test/while/return_closure.lox", "content": "fun f() {\n while (true) {\n var i = \"i\";\n fun g() { print i; }\n return g;\n }\n}\n\nvar h = f();\nh(); // expect: i\n" }, { "path": "test/while/return_inside.lox", "content": "fun f() {\n while (true) {\n var i = \"i\";\n return i;\n }\n}\n\nprint f();\n// expect: i\n" }, { "path": "test/while/syntax.lox", "content": "// Single-expression body.\nvar c = 0;\nwhile (c < 3) print c = c + 1;\n// expect: 1\n// expect: 2\n// expect: 3\n\n// Block body.\nvar a = 0;\nwhile (a < 3) {\n print a;\n a = a + 1;\n}\n// expect: 0\n// expect: 1\n// expect: 2\n\n// Statement bodies.\nwhile (false) if (true) 1; else 2;\nwhile (false) while (true) 1;\nwhile (false) for (;;) 1;\n" }, { "path": "test/while/var_in_body.lox", "content": "// [line 2] Error at 'var': Expect expression.\nwhile (true) var foo;\n" }, { "path": "tool/analysis_options.yaml", "content": "analyzer:\n strong-mode:\n# Close, but still false positives around clamp().\n implicit-casts: false\n# Too many false positives.\n implicit-dynamic: false\n " }, { "path": "tool/bin/benchmark.dart", "content": "import 'dart:convert';\nimport 'dart:io';\n\nimport 'package:path/path.dart' as p;\n\nvoid main(List arguments) {\n if (arguments.isEmpty) {\n print('Usage: benchmark.py [interpreters...] ');\n exit(1);\n }\n\n var interpreters = ['build/clox'];\n var benchmark = arguments.last;\n if (arguments.length > 1) {\n interpreters = arguments.sublist(0, arguments.length - 1);\n }\n\n if (interpreters.length > 1) {\n runComparison(interpreters, benchmark);\n } else {\n runBenchmark(interpreters[0], benchmark);\n }\n}\n\nvoid runBenchmark(String interpreter, String benchmark) {\n var trial = 1;\n var best = 9999.0;\n\n for (;;) {\n var elapsed = runTrial(interpreter, benchmark);\n if (elapsed < best) best = elapsed;\n\n var bestSeconds = best.toStringAsFixed(2);\n print(\"trial #$trial $interpreter best ${bestSeconds}s\");\n trial++;\n }\n}\n\n/// Runs the benchmark once and returns the elapsed time.\ndouble runTrial(String interpreter, String benchmark) {\n var result = Process.runSync(\n interpreter, [p.join(\"test\", \"benchmark\", \"$benchmark.lox\")]);\n var outLines = const LineSplitter().convert(result.stdout as String);\n\n // Remove the trailing last empty line.\n if (outLines.last == \"\") outLines.removeLast();\n\n // The benchmark should print the elapsed time last.\n return double.parse(outLines.last);\n}\n\nvoid runComparison(List interpreters, String benchmark) {\n var trial = 1;\n var best = {for (var interpreter in interpreters) interpreter: 9999.0};\n\n for (;;) {\n for (var interpreter in interpreters) {\n var elapsed = runTrial(interpreter, benchmark);\n if (elapsed < best[interpreter]) best[interpreter] = elapsed;\n }\n\n var bestTime = 999.0;\n var worstTime = 0.0;\n String bestInterpreter;\n for (var interpreter in interpreters) {\n if (best[interpreter] < bestTime) {\n bestTime = best[interpreter];\n bestInterpreter = interpreter;\n }\n if (best[interpreter] > worstTime) {\n worstTime = best[interpreter];\n }\n }\n\n // Turn the time measurement into an effort measurement in units where 1\n // \"work\" is just the total thing the benchmark does.\n var worstWork = 1.0 / worstTime;\n\n print(\"trial #$trial\");\n for (var interpreter in interpreters) {\n String suffix;\n if (interpreter == bestInterpreter) {\n var bestWork = 1.0 / best[interpreter];\n var workRatio = bestWork / worstWork;\n var faster = 100 * (workRatio - 1.0);\n suffix = \"${faster.toStringAsFixed(4)}% faster\";\n } else {\n var ratio = best[interpreter] / bestTime;\n suffix = \"${ratio.toStringAsFixed(4)}x time of best\";\n }\n var bestString = best[interpreter].toStringAsFixed(4);\n print(\" ${interpreter.padRight(30)} best ${bestString}s $suffix\");\n }\n\n trial++;\n }\n}\n" }, { "path": "tool/bin/build.dart", "content": "import 'dart:io';\n\nimport 'package:glob/glob.dart';\nimport 'package:mime_type/mime_type.dart';\nimport 'package:path/path.dart' as p;\nimport 'package:sass/sass.dart' as sass;\nimport 'package:shelf/shelf.dart' as shelf;\nimport 'package:shelf/shelf_io.dart' as io;\n\nimport 'package:tool/src/book.dart';\nimport 'package:tool/src/format.dart';\nimport 'package:tool/src/markdown/markdown.dart';\nimport 'package:tool/src/mustache.dart';\nimport 'package:tool/src/page.dart';\nimport 'package:tool/src/term.dart' as term;\nimport 'package:tool/src/text.dart';\n\n/// Aside comment marker in highlighted code.\nfinal _asideHighlightedCommentPattern =\n RegExp(r' ?// \\[([-a-z0-9]+)\\] *');\n\n/// Aside comment marker in highlighted code with a comment too.\nfinal _asideHighlightedWithCommentPattern =\n RegExp(r' ?// (.+) \\[([-a-z0-9]+)\\] *');\n\n/// Aside comment marker in context lines which are not syntax highlighted.\nfinal _asideCommentPattern = RegExp(r' +// \\[([-a-z0-9]+)\\]');\n\n/// Aside comment marker in context lines which are not syntax highlighted with\n/// a comment too.\nfinal _asideWithCommentPattern = RegExp(r' +// (.+) \\[([-a-z0-9]+)\\]');\n\nFuture main(List arguments) async {\n _buildSass();\n _buildPages();\n\n if (arguments.contains(\"--serve\")) {\n await _runServer();\n }\n}\n\n/// Process each Markdown file.\nvoid _buildPages({bool skipUpToDate = false}) {\n var watch = Stopwatch()..start();\n var book = Book();\n var mustache = Mustache();\n\n DateTime dependenciesModified;\n if (skipUpToDate) {\n dependenciesModified = _mostRecentlyModified(\n [\"asset/mustache/*.html\", \"c/*.{c,h}\", \"java/**.java\"]);\n }\n\n var proseWords = 0;\n var codeLines = 0;\n var totalWords = 0;\n for (var page in book.pages) {\n var metrics = _buildPage(book, mustache, page,\n dependenciesModified: dependenciesModified);\n proseWords += metrics[0];\n codeLines += metrics[1];\n totalWords += metrics[2];\n }\n\n if (totalWords > 0) {\n var seconds = (watch.elapsedMilliseconds / 1000).toStringAsFixed(2);\n print(\"Built ${term.green(proseWords.withCommas)} words and \"\n \"${term.cyan(codeLines.withCommas)} lines of code \"\n \"(${totalWords.withCommas} total words) in $seconds seconds\");\n }\n}\n\nList _buildPage(Book book, Mustache mustache, Page page,\n {DateTime dependenciesModified}) {\n // See if the HTML is up to date.\n if (dependenciesModified != null &&\n _isUpToDate(page.htmlPath, page.markdownPath, dependenciesModified)) {\n return [0, 0, 0];\n }\n\n var proseCount = 0;\n var codeLineCount = 0;\n for (var line in page.lines) proseCount += line.wordCount;\n\n var wordCount = proseCount;\n for (var tag in page.codeTags) {\n var snippet = book.findSnippet(tag);\n if (snippet == null) {\n print(\"No snippet for $tag\");\n continue;\n }\n\n codeLineCount += snippet.added.length;\n for (var line in snippet.added) wordCount += line.wordCount;\n for (var line in snippet.contextBefore) wordCount += line.wordCount;\n for (var line in snippet.contextAfter) wordCount += line.wordCount;\n }\n\n var body = renderMarkdown(book, page, page.lines, Format.web);\n var output = mustache.render(book, page, body);\n\n // Turn aside markers in code into spans. In the empty span case, insert a\n // zero-width space because Chrome seems to lose the span's position if it has\n // no content.\n // // [repl]\n // TODO: Do this directly in the syntax highlighter instead of after the fact.\n output = output.replaceAllMapped(_asideHighlightedCommentPattern,\n (match) => ' ');\n output = output.replaceAllMapped(_asideHighlightedWithCommentPattern,\n (match) => '// ${match[1]}');\n output = output.replaceAllMapped(\n _asideCommentPattern, (match) => ' ');\n output = output.replaceAllMapped(_asideWithCommentPattern,\n (match) => '// ${match[1]}');\n\n // Write the output.\n File(page.htmlPath).writeAsStringSync(output);\n\n var words = \"$wordCount words\";\n if (codeLineCount > 0) words += \", $codeLineCount loc\";\n words = term.gray(\"($words)\");\n\n var number = \"\";\n if (page.numberString.isNotEmpty) {\n number = \"${page.numberString}. \";\n }\n\n if (page.isChapter) {\n print(\" ${term.green('✓')} $number${page.title} $words\");\n } else {\n print(\"${term.green('✓')} $number${page.title} $words\");\n }\n\n return [proseCount, codeLineCount, wordCount];\n}\n\n/// Process each SASS file.\nvoid _buildSass({bool skipUpToDate = false}) {\n var moduleModified = _mostRecentlyModified([\"asset/sass/*.scss\"]);\n\n for (var source in Glob(\"asset/*.scss\").listSync()) {\n var scssPath = p.normalize(source.path);\n var cssPath =\n p.join(\"site\", p.basenameWithoutExtension(source.path) + \".css\");\n\n if (skipUpToDate && _isUpToDate(cssPath, scssPath, moduleModified)) {\n continue;\n }\n\n var output =\n sass.compile(scssPath, color: true, style: sass.OutputStyle.expanded);\n File(cssPath).writeAsStringSync(output);\n print(\"${term.green('-')} $cssPath\");\n }\n}\n\nFuture _runServer() async {\n Future handleRequest(shelf.Request request) async {\n var filePath = p.normalize(p.fromUri(request.url));\n if (filePath == \".\") filePath = \"index.html\";\n var extension = p.extension(filePath).replaceAll(\".\", \"\");\n\n // Refresh files that are being requested.\n if (extension == \"html\") {\n _buildPages(skipUpToDate: true);\n } else if (extension == \"css\") {\n _buildSass(skipUpToDate: true);\n }\n\n try {\n var contents = await File(p.join(\"site\", filePath)).readAsBytes();\n return shelf.Response.ok(contents, headers: {\n HttpHeaders.contentTypeHeader: mimeFromExtension(extension)\n });\n } on FileSystemException {\n print(\n \"${term.red(request.method)} Not found: ${request.url} ($filePath)\");\n return shelf.Response.notFound(\"Could not find '$filePath'.\");\n }\n }\n\n var handler = const shelf.Pipeline().addHandler(handleRequest);\n\n var server = await io.serve(handler, \"localhost\", 8000);\n print(\"Serving at http://${server.address.host}:${server.port}\");\n}\n\n/// Returns `true` if [outputPath] was generated after [inputPath] and more\n/// recently than [dependenciesModified].\nbool _isUpToDate(\n String outputPath, String inputPath, DateTime dependenciesModified) {\n var outputModified = File(outputPath).lastModifiedSync();\n var inputModified = File(inputPath).lastModifiedSync();\n return outputModified.isAfter(dependenciesModified) &&\n outputModified.isAfter(inputModified);\n}\n\n/// The most recently modified time of all files that match [globs].\nDateTime _mostRecentlyModified(List globs) {\n DateTime latest;\n for (var glob in globs) {\n for (var entry in Glob(glob).listSync()) {\n if (entry is File) {\n var modified = entry.lastModifiedSync();\n if (latest == null || modified.isAfter(latest)) latest = modified;\n }\n }\n }\n\n return latest;\n}\n" }, { "path": "tool/bin/build_xml.dart", "content": "import 'dart:io';\n\nimport 'package:path/path.dart' as p;\n\nimport 'package:tool/src/book.dart';\nimport 'package:tool/src/format.dart';\nimport 'package:tool/src/markdown/markdown.dart';\nimport 'package:tool/src/markdown/xml_renderer.dart';\nimport 'package:tool/src/mustache.dart';\nimport 'package:tool/src/page.dart';\nimport 'package:tool/src/term.dart' as term;\n\n/// Generate the XML used to import into InDesign.\n\nFuture main(List arguments) async {\n var book = Book();\n var mustache = Mustache();\n\n await Directory(p.join(\"build\", \"xml\")).create(recursive: true);\n\n for (var page in book.pages) {\n if (!page.isChapter) continue;\n\n if (arguments.isNotEmpty && page.fileName != arguments.first) continue;\n\n _buildPage(book, mustache, page);\n }\n\n // Output a minimal XML file that contains all tags used in the book.\n var allTagsPath = p.join(\"build\", \"xml\", \"all-tags.xml\");\n File(allTagsPath)\n .writeAsStringSync(\"\\n${XmlRenderer.tagFileBuffer}\\n\");\n}\n\nvoid _buildPage(Book book, Mustache mustache, Page page) {\n var xml = renderMarkdown(book, page, page.lines, Format.print);\n\n // Write the output.\n var xmlPath = p.join(\"build\", \"xml\", \"${page.fileName}.xml\");\n File(xmlPath).writeAsStringSync(xml);\n\n print(\"${term.green('-')} ${page.numberString}. ${page.title}\");\n}\n" }, { "path": "tool/bin/compile_snippets.dart", "content": "import 'dart:io';\n\nimport 'package:path/path.dart' as p;\nimport 'package:pool/pool.dart';\n\nimport 'package:tool/src/book.dart';\nimport 'package:tool/src/code_tag.dart';\nimport 'package:tool/src/page.dart';\nimport 'package:tool/src/split_chapter.dart';\nimport 'package:tool/src/term.dart' as term;\n\n/// Tests that various snippets in the middle of chapters can be compiled without\n/// error. Ensures that, as much as possible, we have a working program at\n/// multiple points throughout the chapter.\n\n// TODO: Do this for Java chapters.\n\nvar _chapterTags = >{\n \"Chunks of Bytecode\": [\n \"free-array\",\n \"main-include-chunk\",\n \"simple-instruction\",\n \"add-constant\",\n \"return-after-operand\",\n ],\n \"A Virtual Machine\": [\n \"main-include-vm\",\n \"vm-include-debug\",\n \"print-return\",\n \"main-negate\",\n ],\n \"Scanning on Demand\": [\n \"init-scanner\",\n \"error-token\",\n \"advance\",\n \"match\",\n \"newline\",\n \"peek-next\",\n \"string\",\n \"number\",\n \"identifier-type\",\n \"check-keyword\",\n ],\n \"Compiling Expressions\": [\n \"expression\",\n \"forward-declarations\",\n \"precedence-body\",\n \"infix\",\n \"define-debug-print-code\",\n \"dump-chunk\"\n ],\n \"Types of Values\": [\n \"op-arithmetic\",\n \"print-value\",\n \"disassemble-not\",\n \"values-equal\",\n ],\n \"Strings\": [\n // \"as-string\",\n // We could get things working earlier by moving the \"Operations on Strings\"\n // section before \"Strings\".\n \"value-include-object\",\n \"vm-include-object-memory\",\n ],\n \"Hash Tables\": [\n \"free-table\",\n \"hash-string\",\n \"table-add-all\",\n \"table-get\",\n \"table-delete\",\n \"resize-increment-count\",\n ],\n \"Global Variables\": [\n \"disassemble-print\",\n \"disassemble-pop\",\n \"synchronize\",\n \"define-global-op\",\n \"disassemble-define-global\",\n \"disassemble-get-global\",\n \"disassemble-set-global\",\n ],\n \"Local Variables\": [\n \"local-struct\",\n \"compiler\",\n \"end-scope\",\n \"add-local\",\n \"too-many-locals\",\n \"pop-locals\",\n \"interpret-set-local\",\n ],\n \"Jumping Back and Forth\": [\n \"jump-if-false-op\",\n \"compile-else\",\n \"jump-op\",\n \"pop-end\",\n \"jump-instruction\",\n \"and\",\n \"or\",\n \"while-statement\",\n \"loop-op\",\n \"disassemble-loop\",\n \"for-statement\",\n ],\n \"Calls and Functions\": [\n \"as-function\",\n \"function-type-enum\",\n \"init-compiler\",\n \"init-function-slot\",\n \"return-function\",\n \"disassemble-end\",\n \"runtime-error-temp\",\n \"compile-function\",\n \"init-function-name\",\n \"call\",\n \"interpret\",\n \"disassemble-call\",\n \"return-statement\",\n \"runtime-error-stack\",\n \"return-from-script\",\n \"print-native\",\n \"define-native\",\n \"vm-include-time\"\n ],\n \"Closures\": [\n \"obj-closure\",\n \"new-closure-h\",\n \"print-closure\",\n \"closure-op\",\n \"disassemble-closure\",\n \"interpret-closure\",\n \"runtime-error-function\",\n \"interpret\",\n \"upvalue-struct\",\n \"resolve-upvalue-recurse\",\n \"capture-upvalues\",\n \"debug-include-object\",\n \"obj-upvalue\",\n \"new-upvalue-h\",\n \"print-upvalue\",\n \"upvalue-fields\",\n \"allocate-upvalue-array\",\n \"init-upvalue-fields\",\n \"free-upvalues\",\n \"capture-upvalue\",\n \"interpret-get-upvalue\",\n \"interpret-set-upvalue\",\n \"is-captured-field\",\n \"init-is-captured\",\n \"init-zero-local-is-captured\",\n \"mark-local-captured\",\n \"close-upvalue-op\",\n \"disassemble-close-upvalue\",\n \"next-field\",\n \"init-next\",\n \"open-upvalues-field\",\n \"init-open-upvalues\",\n \"look-for-existing-upvalue\",\n \"insert-upvalue-in-list\",\n \"closed-field\",\n \"init-closed\",\n \"return-close-upvalues\",\n ],\n \"Garbage Collection\": [\n \"collect-garbage-h\",\n \"collect-garbage\",\n \"define-stress-gc\",\n \"call-collect\",\n \"define-log-gc\",\n \"debug-log-includes\",\n \"log-before-collect\",\n \"log-after-collect\",\n \"debug-log-allocate\",\n \"log-free-object\",\n \"mark-value-h\",\n \"mark-object-h\",\n \"is-marked-field\",\n \"init-is-marked\",\n \"log-mark-object\",\n \"mark-table-h\",\n \"mark-table\",\n \"mark-closures\",\n \"mark-open-upvalues\",\n \"memory-include-compiler\",\n \"compiler-include-memory\",\n \"vm-gray-stack\",\n \"init-gray-stack\",\n \"free-gray-stack\",\n \"blacken-closure\",\n \"log-blacken-object\",\n \"check-is-marked\",\n \"sweep\",\n \"unmark\",\n \"table-remove-white-h\",\n \"table-remove-white\",\n \"vm-fields\",\n \"init-gc-fields\",\n \"updated-bytes-allocated\",\n \"collect-on-next\",\n \"heap-grow-factor\",\n \"log-before-size\",\n \"log-collected-amount\",\n \"chunk-include-vm\",\n \"push-string\",\n \"pop-string\",\n \"concatenate-peek\",\n \"concatenate-pop\",\n ],\n \"Classes and Instances\": [\n \"obj-class\",\n \"print-class\",\n \"class-op\",\n \"disassemble-class\",\n \"interpret-class\",\n \"object-include-table\",\n \"print-instance\",\n \"call-class\",\n \"property-ops\",\n \"disassemble-property-ops\",\n \"interpret-get-property\",\n \"get-undefined\",\n \"get-not-instance\",\n \"interpret-set-property\",\n \"set-not-instance\",\n ],\n \"Methods and Initializers\": [\n \"class-methods\",\n \"init-methods\",\n \"free-methods\",\n \"mark-methods\",\n \"method-op\",\n \"disassemble-method\",\n \"define-method\",\n \"obj-bound-method\",\n \"print-bound-method\",\n \"bind-method\",\n \"call-bound-method\",\n \"this\",\n \"slot-zero\",\n \"method-type-enum\",\n \"method-type\",\n \"store-receiver\",\n \"class-compiler-struct\",\n \"create-class-compiler\",\n \"pop-enclosing\",\n \"this-outside-class\",\n \"vm-init-string\",\n \"init-init-string\",\n \"mark-init-string\",\n \"null-init-string\",\n \"clear-init-string\",\n \"initializer-type-enum\",\n \"return-this\",\n \"return-from-init\",\n \"invoke-op\",\n \"invoke-instruction\",\n \"invoke-from-class\",\n \"invoke-field\",\n ],\n \"Superclasses\": [\n \"inherit-op\",\n \"disassemble-inherit\",\n \"interpret-inherit\",\n \"inherit-non-class\",\n \"synthetic-token\",\n \"has-superclass\",\n \"init-has-superclass\",\n \"set-has-superclass\",\n \"get-super-op\",\n \"disassemble-get-super\",\n \"interpret-get-super\",\n \"super-invoke-op\",\n \"disassemble-super-invoke\",\n \"interpret-super-invoke\",\n ],\n \"Optimization\": [\n \"initial-index\",\n \"next-index\",\n \"adjust-alloc\",\n \"adjust-init\",\n \"re-hash\",\n \"adjust-free\",\n \"table-set-grow\",\n \"init-capacity-mask\",\n \"add-all-loop\",\n \"find-string-index\",\n \"find-string-next\",\n \"mark-table\",\n \"remove-white\",\n \"free-table\",\n \"define-nan-boxing\",\n \"end-values-equal\",\n ],\n};\n\nvar _allPassed = true;\n\nFuture main(List arguments) async {\n var watch = Stopwatch()..start();\n var book = Book();\n var pool = Pool(Platform.numberOfProcessors);\n var futures = >[];\n\n for (var chapterName in _chapterTags.keys) {\n var chapter = book.findChapter(chapterName);\n\n var tags = chapter.codeTags;\n var tagNames = _chapterTags[chapterName];\n if (tagNames.isNotEmpty) {\n tags = tagNames.map((name) => book.findTag(chapter, name));\n } else {\n print(\"Warning, no in-chapter snippets for '$chapterName'\");\n }\n\n for (var tag in tags) {\n futures\n .add(pool.withResource(() => _compileChapterTag(book, chapter, tag)));\n }\n }\n\n await Future.wait(futures);\n\n print(\"Done in ${watch.elapsedMilliseconds / 1000} seconds\");\n if (!_allPassed) exit(1);\n}\n\nFuture _compileChapterTag(Book book, Page chapter, CodeTag tag) async {\n await splitChapter(book, chapter, tag);\n\n var buildName = \"${chapter.shortName}-${tag.directory}\";\n var sourceDir = p.join(\"gen\", \"snippets\", chapter.shortName, tag.directory);\n\n var makeArguments = [\n \"-f\",\n \"util/c.make\",\n \"NAME=$buildName\",\n \"MODE=release\",\n \"SOURCE_DIR=$sourceDir\",\n \"SNIPPET=true\"\n ];\n\n var result = await Process.run(\"make\", makeArguments);\n if (result.exitCode == 0) {\n print(\"${term.green('PASS')} ${chapter.title} / ${tag.name}\");\n } else {\n print(\"${term.red('FAIL')} ${chapter.title} / ${tag.name}\");\n print(result.stdout);\n print(result.stderr);\n print(\"\");\n _allPassed = false;\n }\n}\n" }, { "path": "tool/bin/split_chapters.dart", "content": "import 'package:tool/src/book.dart';\nimport 'package:tool/src/split_chapter.dart';\n\nvoid main(List arguments) {\n var book = Book();\n for (var page in book.pages) {\n if (page.language == null) continue;\n splitChapter(book, page);\n }\n}\n" }, { "path": "tool/bin/test.dart", "content": "import 'dart:convert';\nimport 'dart:io';\n\nimport 'package:args/args.dart';\nimport 'package:glob/glob.dart';\nimport 'package:path/path.dart' as p;\n\nimport 'package:tool/src/term.dart' as term;\n\n/// Runs the tests.\n\nfinal _expectedOutputPattern = RegExp(r\"// expect: ?(.*)\");\nfinal _expectedErrorPattern = RegExp(r\"// (Error.*)\");\nfinal _errorLinePattern = RegExp(r\"// \\[((java|c) )?line (\\d+)\\] (Error.*)\");\nfinal _expectedRuntimeErrorPattern = RegExp(r\"// expect runtime error: (.+)\");\nfinal _syntaxErrorPattern = RegExp(r\"\\[.*line (\\d+)\\] (Error.+)\");\nfinal _stackTracePattern = RegExp(r\"\\[line (\\d+)\\]\");\nfinal _nonTestPattern = RegExp(r\"// nontest\");\n\nvar _passed = 0;\nvar _failed = 0;\nvar _skipped = 0;\nvar _expectations = 0;\n\nSuite _suite;\nString _filterPath;\nString _customInterpreter;\nList _customArguments;\n\nfinal _allSuites = {};\nfinal _cSuites = [];\nfinal _javaSuites = [];\n\nclass Suite {\n final String name;\n final String language;\n final String executable;\n final List args;\n final Map tests;\n\n Suite(this.name, this.language, this.executable, this.args, this.tests);\n}\n\nvoid main(List arguments) {\n _defineTestSuites();\n\n var parser = ArgParser();\n\n parser.addOption(\"interpreter\", abbr: \"i\", help: \"Path to interpreter.\");\n parser.addMultiOption(\"arguments\",\n abbr: \"a\", help: \"Additional interpreter arguments.\");\n\n var options = parser.parse(arguments);\n\n if (options.rest.isEmpty) {\n _usageError(parser, \"Missing suite name.\");\n } else if (options.rest.length > 2) {\n _usageError(\n parser, \"Unexpected arguments '${options.rest.skip(2).join(' ')}'.\");\n }\n\n var suite = options.rest[0];\n if (options.rest.length == 2) _filterPath = arguments[1];\n\n if (options.wasParsed(\"interpreter\")) {\n _customInterpreter = options[\"interpreter\"] as String;\n }\n\n if (options.wasParsed(\"arguments\")) {\n _customArguments = options[\"arguments\"] as List;\n\n if (_customInterpreter == null) {\n _usageError(parser,\n \"Must pass an interpreter path if providing custom arguments.\");\n }\n }\n\n if (suite == \"all\") {\n _runSuites(_allSuites.keys.toList());\n } else if (suite == \"c\") {\n _runSuites(_cSuites);\n } else if (suite == \"java\") {\n _runSuites(_javaSuites);\n } else if (!_allSuites.containsKey(suite)) {\n print(\"Unknown interpreter '$suite'\");\n exit(1);\n } else if (!_runSuite(suite)) {\n exit(1);\n }\n}\n\nvoid _usageError(ArgParser parser, String message) {\n print(message);\n print(\"\");\n print(\"Usage: test.dart [filter] [custom interpreter...]\");\n print(\"\");\n print(\"Optional custom interpreter options:\");\n print(parser.usage);\n exit(1);\n}\n\nvoid _runSuites(List names) {\n var anyFailed = false;\n for (var name in names) {\n print(\"=== $name ===\");\n if (!_runSuite(name)) anyFailed = true;\n }\n\n if (anyFailed) exit(1);\n}\n\nbool _runSuite(String name) {\n _suite = _allSuites[name];\n\n _passed = 0;\n _failed = 0;\n _skipped = 0;\n _expectations = 0;\n\n for (var file in Glob(\"test/**.lox\").listSync()) {\n _runTest(file.path);\n }\n\n term.clearLine();\n\n if (_failed == 0) {\n print(\"All ${term.green(_passed)} tests passed \"\n \"($_expectations expectations).\");\n } else {\n print(\"${term.green(_passed)} tests passed. \"\n \"${term.red(_failed)} tests failed.\");\n }\n\n return _failed == 0;\n}\n\nvoid _runTest(String path) {\n if (path.contains(\"benchmark\")) return;\n\n // Make a nice short path relative to the working directory. Normalize it to\n // use \"/\" since the interpreters expect the argument to use that.\n path = p.posix.normalize(path);\n\n // Check if we are just running a subset of the tests.\n if (_filterPath != null) {\n var thisTest = p.posix.relative(path, from: \"test\");\n if (!thisTest.startsWith(_filterPath)) return;\n }\n\n // Update the status line.\n var grayPath = term.gray(\"($path)\");\n term.writeLine(\"Passed: ${term.green(_passed)} \"\n \"Failed: ${term.red(_failed)} \"\n \"Skipped: ${term.yellow(_skipped)} $grayPath\");\n\n // Read the test and parse out the expectations.\n var test = Test(path);\n\n // See if it's a skipped or non-test file.\n if (!test.parse()) return;\n\n var failures = test.run();\n\n // Display the results.\n if (failures.isEmpty) {\n _passed++;\n } else {\n _failed++;\n term.writeLine(\"${term.red(\"FAIL\")} $path\");\n print(\"\");\n for (var failure in failures) {\n print(\" ${term.pink(failure)}\");\n }\n print(\"\");\n }\n}\n\nclass ExpectedOutput {\n final int line;\n final String output;\n\n ExpectedOutput(this.line, this.output);\n}\n\nclass Test {\n final String _path;\n\n final _expectedOutput = [];\n\n /// The set of expected compile error messages.\n final _expectedErrors = {};\n\n /// The expected runtime error message or `null` if there should not be one.\n String _expectedRuntimeError;\n\n /// If there is an expected runtime error, the line it should occur on.\n int _runtimeErrorLine = 0;\n\n int _expectedExitCode = 0;\n\n /// The list of failure message lines.\n final _failures = [];\n\n Test(this._path);\n\n bool parse() {\n // Get the path components.\n var parts = _path.split(\"/\");\n var subpath = \"\";\n String state;\n\n // Figure out the state of the test. We don't break out of this loop because\n // we want lines for more specific paths to override more general ones.\n for (var part in parts) {\n if (subpath.isNotEmpty) subpath += \"/\";\n subpath += part;\n\n if (_suite.tests.containsKey(subpath)) {\n state = _suite.tests[subpath];\n }\n }\n\n if (state == null) {\n throw \"Unknown test state for '$_path'.\";\n } else if (state == \"skip\") {\n _skipped++;\n return false;\n }\n\n var lines = File(_path).readAsLinesSync();\n for (var lineNum = 1; lineNum <= lines.length; lineNum++) {\n var line = lines[lineNum - 1];\n\n // Not a test file at all, so ignore it.\n var match = _nonTestPattern.firstMatch(line);\n if (match != null) return false;\n\n match = _expectedOutputPattern.firstMatch(line);\n if (match != null) {\n _expectedOutput.add(ExpectedOutput(lineNum, match[1]));\n _expectations++;\n continue;\n }\n\n match = _expectedErrorPattern.firstMatch(line);\n if (match != null) {\n _expectedErrors.add(\"[$lineNum] ${match[1]}\");\n\n // If we expect a compile error, it should exit with EX_DATAERR.\n _expectedExitCode = 65;\n _expectations++;\n continue;\n }\n\n match = _errorLinePattern.firstMatch(line);\n if (match != null) {\n // The two interpreters are slightly different in terms of which\n // cascaded errors may appear after an initial compile error because\n // their panic mode recovery is a little different. To handle that,\n // the tests can indicate if an error line should only appear for a\n // certain interpreter.\n var language = match[2];\n if (language == null || language == _suite.language) {\n _expectedErrors.add(\"[${match[3]}] ${match[4]}\");\n\n // If we expect a compile error, it should exit with EX_DATAERR.\n _expectedExitCode = 65;\n _expectations++;\n }\n continue;\n }\n\n match = _expectedRuntimeErrorPattern.firstMatch(line);\n if (match != null) {\n _runtimeErrorLine = lineNum;\n _expectedRuntimeError = match[1];\n // If we expect a runtime error, it should exit with EX_SOFTWARE.\n _expectedExitCode = 70;\n _expectations++;\n }\n }\n\n if (_expectedErrors.isNotEmpty && _expectedRuntimeError != null) {\n print(\"${term.magenta('TEST ERROR')} $_path\");\n print(\" Cannot expect both compile and runtime errors.\");\n print(\"\");\n return false;\n }\n\n // If we got here, it's a valid test.\n return true;\n }\n\n /// Invoke the interpreter and run the test.\n List run() {\n var args = [\n if (_customInterpreter != null) ...?_customArguments else ..._suite.args,\n _path\n ];\n var result = Process.runSync(_customInterpreter ?? _suite.executable, args);\n\n // Normalize Windows line endings.\n var outputLines = const LineSplitter().convert(result.stdout as String);\n var errorLines = const LineSplitter().convert(result.stderr as String);\n\n // Validate that an expected runtime error occurred.\n if (_expectedRuntimeError != null) {\n _validateRuntimeError(errorLines);\n } else {\n _validateCompileErrors(errorLines);\n }\n\n _validateExitCode(result.exitCode, errorLines);\n _validateOutput(outputLines);\n return _failures;\n }\n\n void _validateRuntimeError(List errorLines) {\n if (errorLines.length < 2) {\n fail(\"Expected runtime error '$_expectedRuntimeError' and got none.\");\n return;\n }\n\n if (errorLines[0] != _expectedRuntimeError) {\n fail(\"Expected runtime error '$_expectedRuntimeError' and got:\");\n fail(errorLines[0]);\n }\n\n // Make sure the stack trace has the right line.\n RegExpMatch match;\n var stackLines = errorLines.sublist(1);\n for (var line in stackLines) {\n match = _stackTracePattern.firstMatch(line);\n if (match != null) break;\n }\n\n if (match == null) {\n fail(\"Expected stack trace and got:\", stackLines);\n } else {\n var stackLine = int.parse(match[1]);\n if (stackLine != _runtimeErrorLine) {\n fail(\"Expected runtime error on line $_runtimeErrorLine \"\n \"but was on line $stackLine.\");\n }\n }\n }\n\n void _validateCompileErrors(List error_lines) {\n // Validate that every compile error was expected.\n var foundErrors = {};\n var unexpectedCount = 0;\n for (var line in error_lines) {\n var match = _syntaxErrorPattern.firstMatch(line);\n if (match != null) {\n var error = \"[${match[1]}] ${match[2]}\";\n if (_expectedErrors.contains(error)) {\n foundErrors.add(error);\n } else {\n if (unexpectedCount < 10) {\n fail(\"Unexpected error:\");\n fail(line);\n }\n unexpectedCount++;\n }\n } else if (line != \"\") {\n if (unexpectedCount < 10) {\n fail(\"Unexpected output on stderr:\");\n fail(line);\n }\n unexpectedCount++;\n }\n }\n\n if (unexpectedCount > 10) {\n fail(\"(truncated ${unexpectedCount - 10} more...)\");\n }\n\n // Validate that every expected error occurred.\n for (var error in _expectedErrors.difference(foundErrors)) {\n fail(\"Missing expected error: $error\");\n }\n }\n\n void _validateExitCode(int exitCode, List errorLines) {\n if (exitCode == _expectedExitCode) return;\n\n if (errorLines.length > 10) {\n errorLines = errorLines.sublist(0, 10);\n errorLines.add(\"(truncated...)\");\n }\n\n fail(\"Expected return code $_expectedExitCode and got $exitCode. Stderr:\",\n errorLines);\n }\n\n void _validateOutput(List outputLines) {\n // Remove the trailing last empty line.\n if (outputLines.isNotEmpty && outputLines.last == \"\") {\n outputLines.removeLast();\n }\n\n var index = 0;\n for (; index < outputLines.length; index++) {\n var line = outputLines[index];\n if (index >= _expectedOutput.length) {\n fail(\"Got output '$line' when none was expected.\");\n continue;\n }\n\n var expected = _expectedOutput[index];\n if (expected.output != line) {\n fail(\"Expected output '${expected.output}' on line ${expected.line} \"\n \" and got '$line'.\");\n }\n }\n\n while (index < _expectedOutput.length) {\n var expected = _expectedOutput[index];\n fail(\"Missing expected output '${expected.output}' on line \"\n \"${expected.line}.\");\n index++;\n }\n }\n\n void fail(String message, [List lines]) {\n _failures.add(message);\n if (lines != null) _failures.addAll(lines);\n }\n}\n\nvoid _defineTestSuites() {\n void c(String name, Map tests) {\n var executable = name == \"clox\" ? \"build/cloxd\" : \"build/$name\";\n _allSuites[name] = Suite(name, \"c\", executable, [], tests);\n _cSuites.add(name);\n }\n\n void java(String name, Map tests) {\n var dir = name == \"jlox\" ? \"build/java\" : \"build/gen/$name\";\n _allSuites[name] = Suite(name, \"java\", \"java\",\n [\"-cp\", dir, \"com.craftinginterpreters.lox.Lox\"], tests);\n _javaSuites.add(name);\n }\n\n // These are just for earlier chapters.\n var earlyChapters = {\n \"test/scanning\": \"skip\",\n \"test/expressions\": \"skip\",\n };\n\n // JVM doesn't correctly implement IEEE equality on boxed doubles.\n var javaNaNEquality = {\n \"test/number/nan_equality.lox\": \"skip\",\n };\n\n // No hardcoded limits in jlox.\n var noJavaLimits = {\n \"test/limit/loop_too_large.lox\": \"skip\",\n \"test/limit/no_reuse_constants.lox\": \"skip\",\n \"test/limit/too_many_constants.lox\": \"skip\",\n \"test/limit/too_many_locals.lox\": \"skip\",\n \"test/limit/too_many_upvalues.lox\": \"skip\",\n\n // Rely on JVM for stack overflow checking.\n \"test/limit/stack_overflow.lox\": \"skip\",\n };\n\n // No classes in Java yet.\n var noJavaClasses = {\n \"test/assignment/to_this.lox\": \"skip\",\n \"test/call/object.lox\": \"skip\",\n \"test/class\": \"skip\",\n \"test/closure/close_over_method_parameter.lox\": \"skip\",\n \"test/constructor\": \"skip\",\n \"test/field\": \"skip\",\n \"test/inheritance\": \"skip\",\n \"test/method\": \"skip\",\n \"test/number/decimal_point_at_eof.lox\": \"skip\",\n \"test/number/trailing_dot.lox\": \"skip\",\n \"test/operator/equals_class.lox\": \"skip\",\n \"test/operator/equals_method.lox\": \"skip\",\n \"test/operator/not_class.lox\": \"skip\",\n \"test/regression/394.lox\": \"skip\",\n \"test/super\": \"skip\",\n \"test/this\": \"skip\",\n \"test/return/in_method.lox\": \"skip\",\n \"test/variable/local_from_method.lox\": \"skip\",\n };\n\n // No functions in Java yet.\n var noJavaFunctions = {\n \"test/call\": \"skip\",\n \"test/closure\": \"skip\",\n \"test/for/closure_in_body.lox\": \"skip\",\n \"test/for/return_closure.lox\": \"skip\",\n \"test/for/return_inside.lox\": \"skip\",\n \"test/for/syntax.lox\": \"skip\",\n \"test/function\": \"skip\",\n \"test/operator/not.lox\": \"skip\",\n \"test/regression/40.lox\": \"skip\",\n \"test/return\": \"skip\",\n \"test/unexpected_character.lox\": \"skip\",\n \"test/while/closure_in_body.lox\": \"skip\",\n \"test/while/return_closure.lox\": \"skip\",\n \"test/while/return_inside.lox\": \"skip\",\n };\n\n // No resolution in Java yet.\n var noJavaResolution = {\n \"test/closure/assign_to_shadowed_later.lox\": \"skip\",\n \"test/function/local_mutual_recursion.lox\": \"skip\",\n \"test/variable/collide_with_parameter.lox\": \"skip\",\n \"test/variable/duplicate_local.lox\": \"skip\",\n \"test/variable/duplicate_parameter.lox\": \"skip\",\n \"test/variable/early_bound.lox\": \"skip\",\n\n // Broken because we haven\"t fixed it yet by detecting the error.\n \"test/return/at_top_level.lox\": \"skip\",\n \"test/variable/use_local_in_initializer.lox\": \"skip\",\n };\n\n // No control flow in C yet.\n var noCControlFlow = {\n \"test/block/empty.lox\": \"skip\",\n \"test/for\": \"skip\",\n \"test/if\": \"skip\",\n \"test/limit/loop_too_large.lox\": \"skip\",\n \"test/logical_operator\": \"skip\",\n \"test/variable/unreached_undefined.lox\": \"skip\",\n \"test/while\": \"skip\",\n };\n\n // No functions in C yet.\n var noCFunctions = {\n \"test/call\": \"skip\",\n \"test/closure\": \"skip\",\n \"test/for/closure_in_body.lox\": \"skip\",\n \"test/for/return_closure.lox\": \"skip\",\n \"test/for/return_inside.lox\": \"skip\",\n \"test/for/syntax.lox\": \"skip\",\n \"test/function\": \"skip\",\n \"test/limit/no_reuse_constants.lox\": \"skip\",\n \"test/limit/stack_overflow.lox\": \"skip\",\n \"test/limit/too_many_constants.lox\": \"skip\",\n \"test/limit/too_many_locals.lox\": \"skip\",\n \"test/limit/too_many_upvalues.lox\": \"skip\",\n \"test/regression/40.lox\": \"skip\",\n \"test/return\": \"skip\",\n \"test/unexpected_character.lox\": \"skip\",\n \"test/variable/collide_with_parameter.lox\": \"skip\",\n \"test/variable/duplicate_parameter.lox\": \"skip\",\n \"test/variable/early_bound.lox\": \"skip\",\n \"test/while/closure_in_body.lox\": \"skip\",\n \"test/while/return_closure.lox\": \"skip\",\n \"test/while/return_inside.lox\": \"skip\",\n };\n\n // No classes in C yet.\n var noCClasses = {\n \"test/assignment/to_this.lox\": \"skip\",\n \"test/call/object.lox\": \"skip\",\n \"test/class\": \"skip\",\n \"test/closure/close_over_method_parameter.lox\": \"skip\",\n \"test/constructor\": \"skip\",\n \"test/field\": \"skip\",\n \"test/inheritance\": \"skip\",\n \"test/method\": \"skip\",\n \"test/number/decimal_point_at_eof.lox\": \"skip\",\n \"test/number/trailing_dot.lox\": \"skip\",\n \"test/operator/equals_class.lox\": \"skip\",\n \"test/operator/equals_method.lox\": \"skip\",\n \"test/operator/not.lox\": \"skip\",\n \"test/operator/not_class.lox\": \"skip\",\n \"test/regression/394.lox\": \"skip\",\n \"test/return/in_method.lox\": \"skip\",\n \"test/super\": \"skip\",\n \"test/this\": \"skip\",\n \"test/variable/local_from_method.lox\": \"skip\",\n };\n\n // No inheritance in C yet.\n var noCInheritance = {\n \"test/class/local_inherit_other.lox\": \"skip\",\n \"test/class/local_inherit_self.lox\": \"skip\",\n \"test/class/inherit_self.lox\": \"skip\",\n \"test/class/inherited_method.lox\": \"skip\",\n \"test/inheritance\": \"skip\",\n \"test/regression/394.lox\": \"skip\",\n \"test/super\": \"skip\",\n };\n\n java(\"jlox\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...javaNaNEquality,\n ...noJavaLimits,\n });\n\n java(\"chap04_scanning\", {\n // No interpreter yet.\n \"test\": \"skip\",\n \"test/scanning\": \"pass\"\n });\n\n // No test for chapter 5. It just has a hardcoded main() in AstPrinter.\n\n java(\"chap06_parsing\", {\n // No real interpreter yet.\n \"test\": \"skip\",\n \"test/expressions/parse.lox\": \"pass\"\n });\n\n java(\"chap07_evaluating\", {\n // No real interpreter yet.\n \"test\": \"skip\",\n \"test/expressions/evaluate.lox\": \"pass\"\n });\n\n java(\"chap08_statements\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...javaNaNEquality,\n ...noJavaLimits,\n ...noJavaFunctions,\n ...noJavaResolution,\n ...noJavaClasses,\n\n // No control flow.\n \"test/block/empty.lox\": \"skip\",\n \"test/for\": \"skip\",\n \"test/if\": \"skip\",\n \"test/logical_operator\": \"skip\",\n \"test/while\": \"skip\",\n \"test/variable/unreached_undefined.lox\": \"skip\",\n });\n\n java(\"chap09_control\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...javaNaNEquality,\n ...noJavaLimits,\n ...noJavaFunctions,\n ...noJavaResolution,\n ...noJavaClasses,\n });\n\n java(\"chap10_functions\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...javaNaNEquality,\n ...noJavaLimits,\n ...noJavaResolution,\n ...noJavaClasses,\n });\n\n java(\"chap11_resolving\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...javaNaNEquality,\n ...noJavaLimits,\n ...noJavaClasses,\n });\n\n java(\"chap12_classes\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...noJavaLimits,\n ...javaNaNEquality,\n\n // No inheritance.\n \"test/class/local_inherit_other.lox\": \"skip\",\n \"test/class/local_inherit_self.lox\": \"skip\",\n \"test/class/inherit_self.lox\": \"skip\",\n \"test/class/inherited_method.lox\": \"skip\",\n \"test/inheritance\": \"skip\",\n \"test/regression/394.lox\": \"skip\",\n \"test/super\": \"skip\",\n });\n\n java(\"chap13_inheritance\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...javaNaNEquality,\n ...noJavaLimits,\n });\n\n c(\"clox\", {\n \"test\": \"pass\",\n ...earlyChapters,\n });\n\n c(\"chap17_compiling\", {\n // No real interpreter yet.\n \"test\": \"skip\",\n \"test/expressions/evaluate.lox\": \"pass\",\n });\n\n c(\"chap18_types\", {\n // No real interpreter yet.\n \"test\": \"skip\",\n \"test/expressions/evaluate.lox\": \"pass\",\n });\n\n c(\"chap19_strings\", {\n // No real interpreter yet.\n \"test\": \"skip\",\n \"test/expressions/evaluate.lox\": \"pass\",\n });\n\n c(\"chap20_hash\", {\n // No real interpreter yet.\n \"test\": \"skip\",\n \"test/expressions/evaluate.lox\": \"pass\",\n });\n\n c(\"chap21_global\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...noCControlFlow,\n ...noCFunctions,\n ...noCClasses,\n\n // No blocks.\n \"test/assignment/local.lox\": \"skip\",\n \"test/variable/in_middle_of_block.lox\": \"skip\",\n \"test/variable/in_nested_block.lox\": \"skip\",\n \"test/variable/scope_reuse_in_different_blocks.lox\": \"skip\",\n \"test/variable/shadow_and_local.lox\": \"skip\",\n \"test/variable/undefined_local.lox\": \"skip\",\n\n // No local variables.\n \"test/block/scope.lox\": \"skip\",\n \"test/variable/duplicate_local.lox\": \"skip\",\n \"test/variable/shadow_global.lox\": \"skip\",\n \"test/variable/shadow_local.lox\": \"skip\",\n \"test/variable/use_local_in_initializer.lox\": \"skip\",\n });\n\n c(\"chap22_local\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...noCControlFlow,\n ...noCFunctions,\n ...noCClasses,\n });\n\n c(\"chap23_jumping\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...noCFunctions,\n ...noCClasses,\n });\n\n c(\"chap24_calls\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...noCClasses,\n\n // No closures.\n \"test/closure\": \"skip\",\n \"test/for/closure_in_body.lox\": \"skip\",\n \"test/for/return_closure.lox\": \"skip\",\n \"test/function/local_recursion.lox\": \"skip\",\n \"test/limit/too_many_upvalues.lox\": \"skip\",\n \"test/regression/40.lox\": \"skip\",\n \"test/while/closure_in_body.lox\": \"skip\",\n \"test/while/return_closure.lox\": \"skip\",\n });\n\n c(\"chap25_closures\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...noCClasses,\n });\n\n c(\"chap26_garbage\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...noCClasses,\n });\n\n c(\"chap27_classes\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...noCInheritance,\n\n // No methods.\n \"test/assignment/to_this.lox\": \"skip\",\n \"test/class/local_reference_self.lox\": \"skip\",\n \"test/class/reference_self.lox\": \"skip\",\n \"test/closure/close_over_method_parameter.lox\": \"skip\",\n \"test/constructor\": \"skip\",\n \"test/field/get_and_set_method.lox\": \"skip\",\n \"test/field/method.lox\": \"skip\",\n \"test/field/method_binds_this.lox\": \"skip\",\n \"test/method\": \"skip\",\n \"test/operator/equals_class.lox\": \"skip\",\n \"test/operator/equals_method.lox\": \"skip\",\n \"test/return/in_method.lox\": \"skip\",\n \"test/this\": \"skip\",\n \"test/variable/local_from_method.lox\": \"skip\",\n });\n\n c(\"chap28_methods\", {\n \"test\": \"pass\",\n ...earlyChapters,\n ...noCInheritance,\n });\n\n c(\"chap29_superclasses\", {\n \"test\": \"pass\",\n ...earlyChapters,\n });\n\n c(\"chap30_optimization\", {\n \"test\": \"pass\",\n ...earlyChapters,\n });\n}\n" }, { "path": "tool/bin/tile_pages.dart", "content": "import 'dart:io';\n\nimport 'package:image/image.dart';\nimport 'package:path/path.dart' as p;\n\n/// Convert a PDF to a tiled PNG image of all of the pages.\n///\n/// Requires `pdftoppm` which can be installed on Mac with:\n///\n/// brew install poppler\nFuture main(List arguments) async {\n print('Exporting PDF pages to PNG...');\n var tempDir = await Directory('.').createTemp('pages');\n\n // The `-r` argument is DPI.\n var result = await Process.run('pdftoppm',\n ['-png', '-r', '40', arguments[0], p.join(tempDir.path, 'page')]);\n if (result.exitCode != 0) {\n print('Could not export pages:\\n${result.stdout}\\n${result.stderr}');\n }\n\n var pages = [];\n var imageFiles = tempDir\n .listSync()\n .whereType()\n .where((entry) => entry.path.endsWith('.png'))\n .toList();\n imageFiles.sort((a, b) => a.path.compareTo(b.path));\n\n for (var imageFile in imageFiles) {\n print('Reading ${imageFile.path}...');\n var bytes = await imageFile.readAsBytes();\n pages.add(decodePng(bytes));\n }\n\n const columns = 36;\n const rows = 18;\n const border = 4;\n\n var pageWidth = pages.first.width;\n var pageHeight = pages.first.height;\n\n var tiled = Image.rgb((pageWidth + border) * columns + border,\n (pageHeight + border) * rows + border);\n tiled.fill(Color.fromRgb(0, 0, 0));\n\n for (var i = 0; i < pages.length; i++) {\n var x = i % columns;\n var y = i ~/ columns;\n print('Tiling page ${i + 1} ($x, $y)...');\n copyInto(tiled, pages[i],\n dstX: x * (pageWidth + border) + border,\n dstY: y * (pageHeight + border) + border);\n }\n\n print('Writing pages.png...');\n await File('pages.png').writeAsBytes(encodePng(tiled));\n\n await tempDir.delete(recursive: true);\n}\n" }, { "path": "tool/lib/src/book.dart", "content": "import 'code_tag.dart';\nimport 'location.dart';\nimport 'page.dart';\nimport 'snippet.dart';\nimport 'source_file_parser.dart';\nimport 'text.dart';\n\nimport 'package:glob/glob.dart';\nimport 'package:path/path.dart' as p;\n\nconst _tableOfContents = {\n '': [\n 'Crafting Interpreters',\n 'Dedication',\n 'Acknowledgements',\n 'Table of Contents',\n ],\n 'Welcome': [\n 'Introduction',\n 'A Map of the Territory',\n 'The Lox Language',\n ],\n 'A Tree-Walk Interpreter': [\n 'Scanning',\n 'Representing Code',\n 'Parsing Expressions',\n 'Evaluating Expressions',\n 'Statements and State',\n 'Control Flow',\n 'Functions',\n 'Resolving and Binding',\n 'Classes',\n 'Inheritance',\n ],\n 'A Bytecode Virtual Machine': [\n 'Chunks of Bytecode',\n 'A Virtual Machine',\n 'Scanning on Demand',\n 'Compiling Expressions',\n 'Types of Values',\n 'Strings',\n 'Hash Tables',\n 'Global Variables',\n 'Local Variables',\n 'Jumping Back and Forth',\n 'Calls and Functions',\n 'Closures',\n 'Garbage Collection',\n 'Classes and Instances',\n 'Methods and Initializers',\n 'Superclasses',\n 'Optimization',\n ],\n 'Backmatter': [\n 'Appendix I',\n 'Appendix II',\n ],\n};\n\n/// The contents of the Markdown and source files for the book, loaded and\n/// parsed.\nclass Book {\n final List parts = [];\n final List frontmatter = [];\n final List pages = [];\n\n final Map _snippets = {};\n\n Book() {\n var partIndex = 1;\n var chapterIndex = 1;\n var inMatter = false;\n\n // Load the pages.\n for (var part in _tableOfContents.keys) {\n // Front- and backmatter have no names, pages, or numbers.\n var partNumber = \"\";\n inMatter = part == \"\" || part == \"Backmatter\";\n if (!inMatter) {\n partNumber = partIndex.roman;\n partIndex += 1;\n }\n\n // There is no part page for the frontmatter.\n Page partPage;\n if (part != \"\") {\n partPage = Page(part, null, partNumber, pages.length);\n pages.add(partPage);\n parts.add(partPage);\n }\n\n for (var chapter in _tableOfContents[part]) {\n var chapterNumber = \"\";\n if (inMatter) {\n // Front- and backmatter chapters are specially numbered.\n if (chapter == \"Appendix I\") {\n chapterNumber = \"A1\";\n } else if (chapter == \"Appendix II\") {\n chapterNumber = \"A2\";\n }\n } else {\n chapterNumber = chapterIndex.toString();\n chapterIndex++;\n }\n\n var page = Page(chapter, partPage, chapterNumber, pages.length);\n pages.add(page);\n if (partPage != null) {\n partPage.chapters.add(page);\n } else {\n frontmatter.add(page);\n }\n }\n }\n\n // Load the source files.\n for (var language in [\"java\", \"c\"]) {\n for (var file in Glob(\"$language/**.{c,h,java}\").listSync()) {\n var shortPath = p.relative(file.path, from: language);\n var sourceFile = SourceFileParser(this, file.path, shortPath).parse();\n\n // Create snippets from the lines in the file.\n var lineIndex = 0;\n for (var line in sourceFile.lines) {\n var snippet = _snippets.putIfAbsent(\n line.start, () => Snippet(sourceFile, line.start));\n snippet.addLine(lineIndex, line);\n\n if (line.end != null) {\n var endSnippet = _snippets.putIfAbsent(\n line.end, () => Snippet(sourceFile, line.end));\n endSnippet.removeLine(lineIndex, line);\n }\n\n lineIndex++;\n }\n }\n }\n\n for (var snippet in _snippets.values) {\n if (snippet.tag.name == \"not-yet\") continue;\n if (snippet.tag.name == \"omit\") continue;\n snippet.calculateContext();\n }\n }\n\n /// Looks for a page with [title].\n Page findChapter(String title) =>\n pages.firstWhere((page) => page.title == title);\n\n /// Looks for a page with [number];\n Page findNumber(String number) =>\n pages.firstWhere((page) => page.numberString == number);\n\n /// Gets the [Page] [offset] pages before or after this one.\n Page adjacentPage(Page start, int offset) {\n var index = pages.indexOf(start) + offset;\n if (index < 0 || index >= pages.length) return null;\n return pages[index];\n }\n\n Snippet findSnippet(CodeTag tag) => _snippets[tag];\n\n /// Gets the last snippet that appears in [page].\n ///\n /// Note: Not very fast.\n Snippet lastSnippet(Page page) {\n Snippet last;\n for (var snippet in _snippets.values) {\n if (snippet.tag.chapter != page) continue;\n if (last == null || snippet.tag > last.tag) last = snippet;\n }\n\n return last;\n }\n\n /// Find the [CodeTag] with [name] on [page].\n ///\n /// Note: Not very fast.\n CodeTag findTag(Page page, String name) {\n for (var tag in _snippets.keys) {\n if (tag.chapter != page) continue;\n if (tag.name == name) return tag;\n }\n\n throw ArgumentError(\"Could not find tag '$name' in '${page.title}'.\");\n }\n}\n\n/// A single source file whose code is included in the book.\nclass SourceFile {\n final String path;\n final List lines = [];\n\n SourceFile(this.path);\n\n String get language => path.endsWith(\"java\") ? \"java\" : \"c\";\n\n String get nicePath => path.replaceAll(\"com/craftinginterpreters/\", \"\");\n}\n\n/// A line of code in a [SourceFile] and the metadata for it.\nclass SourceLine {\n final String text;\n final Location location;\n\n /// The first snippet where this line appears in the book.\n final CodeTag start;\n\n /// The last snippet where this line is removed, or null if the line reaches\n /// the end of the book.\n final CodeTag end;\n\n SourceLine(this.text, this.location, this.start, this.end);\n\n /// Returns true if this line exists by the time we reach [tag].\n bool isPresent(CodeTag tag) {\n // If we haven't reached this line's snippet yet.\n if (tag < start) return false;\n\n // If we are past the snippet where it is removed.\n if (end != null && tag >= end) return false;\n\n return true;\n }\n\n String toString() {\n var result = \"${text.padRight(72)} // $start\";\n if (end != null) result += \" < $end\";\n return result;\n }\n}\n" }, { "path": "tool/lib/src/code_tag.dart", "content": "import 'page.dart';\n\nclass CodeTag with Ordering implements Comparable {\n final Page chapter;\n final String name;\n\n /// The zero-based index of the tag in the order that it appears on the page.\n final int _index;\n\n /// Number of preceding lines of context to show.\n final int beforeCount;\n\n /// Number of trailing lines of context to show.\n final int afterCount;\n\n /// Whether to show location information.\n final bool showLocation;\n\n factory CodeTag(Page chapter, String name, int index, int beforeCount,\n int afterCount, bool showLocation) {\n // Hackish. Always want \"not-yet\" to be the last tag even if it appears\n // before a real tag. That ensures we can push it for other tags that have\n // been named.\n if (name == \"not-yet\") index = 9999;\n\n return CodeTag._(\n chapter, name, index, beforeCount, afterCount, showLocation);\n }\n\n CodeTag._(this.chapter, this.name, this._index, this.beforeCount,\n this.afterCount, this.showLocation);\n\n /// Gets the name of the directory used for this tag when the code is split\n /// at this tag's snippet.\n String get directory {\n var index = _index.toString().padLeft(2, \"0\");\n return \"$index-$name\";\n }\n\n int compareTo(CodeTag other) {\n if (chapter.ordinal != other.chapter.ordinal) {\n return chapter.ordinal.compareTo(other.chapter.ordinal);\n }\n\n return _index.compareTo(other._index);\n }\n\n String toString() => \"Tag(${chapter.ordinal}|$_index: $chapter $name)\";\n}\n\n/// Implements the comparison operators in terms of [compareTo()].\nmixin Ordering implements Comparable {\n bool operator <(T other) => compareTo(other) < 0;\n bool operator <=(T other) => compareTo(other) <= 0;\n bool operator >(T other) => compareTo(other) > 0;\n bool operator >=(T other) => compareTo(other) >= 0;\n}\n" }, { "path": "tool/lib/src/format.dart", "content": "/// The book format being rendered to.\nenum Format {\n /// HTML for the web.\n web,\n\n /// XML for importing into InDesign.\n print,\n}\n\nextension FormatExtension on Format {\n bool get isWeb => this == Format.web;\n bool get isPrint => this == Format.print;\n}\n" }, { "path": "tool/lib/src/location.dart", "content": "/// The context in which a line of code appears. The chain of types and\n/// functions it's in.\nclass Location {\n final Location parent;\n final String kind;\n String _name;\n final String signature;\n\n /// If [kind] is \"method\" or \"function\" then this tracks where we are\n /// declaring or defining the function.\n final bool isFunctionDeclaration;\n\n Location(this.parent, this.kind, this._name,\n {this.signature, this.isFunctionDeclaration = false});\n\n String get name => _name;\n\n set name(String value) {\n // Can only set the name if it's an unnamed typedef.\n assert(_name == null);\n _name = value;\n }\n\n bool get isFile => kind == \"file\";\n\n bool get isFunction =>\n const {\"constructor\", \"function\", \"method\"}.contains(kind);\n\n int get depth {\n var current = this;\n var result = 0;\n while (current != null) {\n result++;\n current = current.parent;\n }\n return result;\n }\n\n String toString() {\n var result = \"$kind $name\";\n if (signature != null) result += \"($signature)\";\n if (parent != null) result = \"$parent > $result\";\n return result;\n }\n\n /// Generates a string of HTML that describes a snippet at this location,\n /// when following the [preceding] location.\n String toHtml(Location preceding, List removed) {\n if (kind == \"new\") return \"create new file\";\n if (kind == \"top\") return \"add to top of file\";\n\n // Note: The order of these is highly significant.\n if (kind == \"class\" && parent?.kind == \"class\") {\n return \"nest inside class ${parent.name}\";\n }\n\n if (isFunction && preceding == this) {\n // Hack. There's one place where we add a new overload and that shouldn't\n // be treated as in the same function. But we can't always look at the\n // signature because there's another place where a change signature would\n // confuse the build script. So just check for the one-off case here.\n if (name == \"resolve\" && signature == \"Expr expr\") {\n return \"add after ${preceding.name}(${preceding.signature})\";\n }\n\n // We're still inside a function.\n return \"in $name()\";\n }\n\n if (isFunction && removed.isNotEmpty) {\n // Hack. We don't appear to be in the middle of a function, but we are\n // replacing lines, so assume we're replacing the entire function.\n return \"$kind $name()\";\n }\n\n if (parent == preceding && !preceding.isFile) {\n // We're nested inside a type.\n return \"in ${preceding.kind} ${preceding.name}\";\n }\n\n if (preceding == this && !isFile) {\n // We're still inside a type.\n return \"in $kind $name\";\n }\n\n if (preceding.isFunction) {\n // We aren't inside a function, but we do know the preceding one.\n return \"add after ${preceding.name}()\";\n }\n\n if (!preceding.isFile) {\n // We aren't inside any function, but we do know what we follow.\n return \"add after ${preceding.kind} ${preceding.name}\";\n }\n\n // If we get here, there isn't a useful location to show. The snippet will\n // have enough surrounding context to make it clear. This is usually stuff\n // like imports or includes near the top of the file.\n return null;\n }\n\n /// Generates a string of InDesign XML that describes a snippet at this\n /// location, when following the [preceding] location.\n ///\n /// This is similar to [toHtml] but uses different tags and places the\n /// signatures inside the tags instead of outside.\n String toXml(Location preceding, List removed) {\n if (kind == \"new\") return \"create new file\";\n if (kind == \"top\") return \"add to top of file\";\n\n // Note: The order of these is highly significant.\n if (kind == \"class\" && parent?.kind == \"class\") {\n return \"nest inside class ${parent.name}\";\n }\n\n if (isFunction && preceding == this) {\n // Hack. There's one place where we add a new overload and that shouldn't\n // be treated as in the same function. But we can't always look at the\n // signature because there's another place where a change signature would\n // confuse the build script. So just check for the one-off case here.\n if (name == \"resolve\" && signature == \"Expr expr\") {\n return \"add after ${preceding.name}\"\n \"(${preceding.signature})\";\n }\n\n // We're still inside a function.\n return \"in $name()\";\n }\n\n if (isFunction && removed.isNotEmpty) {\n // Hack. We don't appear to be in the middle of a function, but we are\n // replacing lines, so assume we're replacing the entire function.\n return \"$kind $name()\";\n }\n\n if (parent == preceding && !preceding.isFile) {\n // We're nested inside a type.\n return \"in ${preceding.kind} \"\n \"${preceding.name}\";\n }\n\n if (preceding == this && !isFile) {\n // We're still inside a type.\n return \"in $kind $name\";\n }\n\n if (preceding.isFunction) {\n // We aren't inside a function, but we do know the preceding one.\n return \"add after ${preceding.name}()\";\n }\n\n if (!preceding.isFile) {\n // We aren't inside any function, but we do know what we follow.\n if (preceding.isFunction) {\n return \"add after ${preceding.kind} \"\n \"${preceding.name}()\";\n } else {\n return \"add after ${preceding.kind} \"\n \"${preceding.name}\";\n }\n }\n\n // If we get here, there isn't a useful location to show. The snippet will\n // have enough surrounding context to make it clear. This is usually stuff\n // like imports or includes near the top of the file.\n return null;\n }\n\n bool operator ==(Object other) {\n // Note: Signature is deliberately not considered part of equality. There's\n // a case in calls-and-functions where the signature of a function changes\n // and it confuses the build script if we treat the signatures as\n // significant.\n return other is Location && kind == other.kind && name == other.name;\n }\n\n int get hashCode => kind.hashCode ^ name.hashCode;\n\n /// Discard as many children as needed to get to [depth] parents.\n Location popToDepth(int depth) {\n var current = this;\n var locations = [];\n while (current != null) {\n locations.add(current);\n current = current.parent;\n }\n\n // If we are already shallower, there is nothing to pop.\n if (locations.length < depth + 1) return this;\n\n return locations[locations.length - depth - 1];\n }\n}\n" }, { "path": "tool/lib/src/markdown/block_syntax.dart", "content": "import 'package:markdown/markdown.dart';\n\nimport '../format.dart';\nimport '../page.dart';\n\n/// Parses atx-style headers like `## Header` and gives them the book's special\n/// handling:\n///\n/// - Generates anchor links.\n/// - Includes the section numbers.\nclass BookHeaderSyntax extends BlockSyntax {\n /// Leading `#` define atx-style headers.\n static final _headerPattern = RegExp(r'^(#{1,6}) (.*)$');\n\n final Page _page;\n final Format _format;\n\n RegExp get pattern => _headerPattern;\n\n BookHeaderSyntax(this._page, this._format);\n\n Node parse(BlockParser parser) {\n var header = _page.headers[parser.current];\n parser.advance();\n\n if (_format.isPrint) {\n return Element(\"h${header.level}\", [UnparsedContent(header.name)]);\n }\n\n var number = \"\";\n if (!header.isSpecial) {\n number = \"${_page.numberString} . ${header.headerIndex}\";\n if (header.subheaderIndex != null) {\n number += \" . ${header.subheaderIndex}\";\n }\n }\n\n var link = Element(\"a\", [\n if (!header.isSpecial) Element(\"small\", [Text(number)]),\n UnparsedContent(header.name)\n ]);\n link.attributes[\"href\"] = \"#${header.anchor}\";\n link.attributes[\"id\"] = header.anchor;\n\n return Element(\"h${header.level}\", [link]);\n }\n}\n" }, { "path": "tool/lib/src/markdown/code_syntax.dart", "content": "import 'package:markdown/markdown.dart';\n\nimport '../book.dart';\nimport '../code_tag.dart';\nimport '../format.dart';\nimport '../page.dart';\nimport '../snippet.dart';\nimport '../syntax/highlighter.dart';\nimport '../text.dart';\n\n/// Custom code block formatter that uses our syntax highlighter.\nclass HighlightedCodeBlockSyntax extends BlockSyntax {\n static final _codeFencePattern = RegExp(r'^(\\s*)```(.*)$');\n\n final Format _format;\n\n RegExp get pattern => _codeFencePattern;\n\n HighlightedCodeBlockSyntax(this._format);\n\n bool canParse(BlockParser parser) =>\n pattern.firstMatch(parser.current) != null;\n\n List parseChildLines(BlockParser parser) {\n var childLines = [];\n parser.advance();\n\n while (!parser.isDone) {\n var match = pattern.firstMatch(parser.current);\n if (match == null) {\n childLines.add(parser.current);\n parser.advance();\n } else {\n parser.advance();\n break;\n }\n }\n\n return childLines;\n }\n\n Node parse(BlockParser parser) {\n // Get the syntax identifier, if there is one.\n var match = pattern.firstMatch(parser.current);\n var indent = match[1].length;\n var language = match[2];\n\n var childLines = parseChildLines(parser);\n\n String code;\n if (language == \"text\") {\n // Don't syntax highlight text.\n var buffer = StringBuffer();\n if (!_format.isPrint) {\n buffer.write(\"
\");\n\n        // The HTML spec mandates that a leading newline after '
' is\n        // ignored.\n        // https://html.spec.whatwg.org/#element-restrictions\n        // Some snippets deliberately start with a newline which needs to be\n        // preserved, so output an extra (discarded) newline in that case.\n        if (_format.isWeb && childLines.first.isEmpty) buffer.writeln();\n      }\n\n      for (var line in childLines) {\n        // Strip off any leading indentation.\n        if (line.length > indent) line = line.substring(indent);\n        checkLineLength(line);\n\n        buffer.write(line.escapeHtml);\n        if (_format.isPrint) {\n          // Soft break, so that the code stays one paragraph.\n          buffer.write(\"
\");\n        } else {\n          buffer.writeln();\n        }\n      }\n\n      if (!_format.isPrint) buffer.write(\"
\");\n\n      code = buffer.toString();\n    } else {\n      code = formatCode(language, childLines, _format, indent: indent);\n    }\n\n    if (_format.isPrint) {\n      // Remove the trailing newline since we'll write a newline after the\n      // \"
\" and we don't want InDesign to insert a blank paragraph.\n code = code.trimTrailingNewline();\n\n // Replace newlines with soft breaks so that InDesign treats the entire\n // snippet as a single paragraph and keeps it together.\n code = code.replaceAll(\"\\n\", \"
\");\n\n // Don't wrap in a div for XML.\n return Element.text(\"pre\", code);\n }\n\n var element = Element.text(\"div\", code);\n element.attributes[\"class\"] = \"codehilite\";\n return element;\n }\n}\n\n/// Recognizes `^code` tags and inserts the relevant snippet.\nclass CodeTagBlockSyntax extends BlockSyntax {\n static final _startPattern = RegExp(r'\\^code ([a-z0-9-]+)');\n\n final Book _book;\n final Page _page;\n final Format _format;\n\n CodeTagBlockSyntax(this._book, this._page, this._format);\n\n RegExp get pattern => _startPattern;\n\n bool canParse(BlockParser parser) =>\n pattern.firstMatch(parser.current) != null;\n\n Node parse(BlockParser parser) {\n var match = pattern.firstMatch(parser.current);\n var name = match[1];\n parser.advance();\n\n var codeTag = _page.findCodeTag(name);\n String snippet;\n if (_format.isPrint) {\n snippet = _buildSnippetXml(codeTag, _book.findSnippet(codeTag));\n } else {\n snippet = _buildSnippet(_format, codeTag, _book.findSnippet(codeTag));\n }\n return Text(snippet);\n }\n}\n\nString _buildSnippet(Format format, CodeTag tag, Snippet snippet) {\n // NOTE: If you change this, be sure to update the baked in example snippet\n // in introduction.md.\n\n if (snippet == null) {\n print(\"Undefined snippet ${tag.name}\");\n return \"ERROR: Missing snippet ${tag.name}\\n\";\n }\n\n var location = [];\n if (tag.showLocation) location = snippet.locationHtmlLines;\n\n var buffer = StringBuffer();\n buffer.write('
');\n\n if (snippet.contextBefore.isNotEmpty) {\n _writeContextHtml(format, buffer, snippet.contextBefore,\n cssClass: snippet.added.isNotEmpty ? \"insert-before\" : null);\n }\n\n if (snippet.addedComma != null) {\n var commaLine = formatCode(\n snippet.file.language, [snippet.addedComma], format,\n preClass: \"insert-before\");\n var comma = commaLine.lastIndexOf(\",\");\n buffer.write(commaLine.substring(0, comma));\n buffer.write(',');\n buffer.write(commaLine.substring(comma + 1));\n }\n\n if (tag.showLocation) {\n var lines = location.join(\"
\\n\");\n buffer.writeln('
$lines
');\n }\n\n if (snippet.added != null) {\n var added = formatCode(snippet.file.language, snippet.added, format,\n preClass: tag.beforeCount > 0 || tag.afterCount > 0 ? \"insert\" : null);\n buffer.write(added);\n }\n\n if (snippet.contextAfter.isNotEmpty) {\n _writeContextHtml(format, buffer, snippet.contextAfter,\n cssClass: snippet.added.isNotEmpty ? \"insert-after\" : null);\n }\n\n buffer.writeln('
');\n\n if (tag.showLocation) {\n var lines = location.join(\", \");\n buffer.writeln('
$lines
');\n }\n\n return buffer.toString();\n}\n\nString _buildSnippetXml(CodeTag tag, Snippet snippet) {\n var buffer = StringBuffer();\n\n if (tag.showLocation) buffer.writeln(snippet.locationXml);\n\n if (snippet.contextBefore.isNotEmpty) {\n _writeContextXml(buffer, snippet.contextBefore, \"before\");\n }\n\n if (snippet.addedComma != null) {\n // TODO: How should this look in print?\n buffer.write(\"TODO added comma\");\n// var commaLine = formatCode(snippet.file.language, [snippet.addedComma],\n// preClass: \"insert-before\", xml: true);\n// var comma = commaLine.lastIndexOf(\",\");\n// buffer.write(commaLine.substring(0, comma));\n// buffer.write(',');\n// buffer.write(commaLine.substring(comma + 1));\n }\n\n if (snippet.added != null) {\n // Use different tags based on whether there is context before, after,\n // neither, or both.\n String insertTag;\n if (tag.beforeCount > 0) {\n if (tag.afterCount > 0) {\n insertTag = \"interpreter-between\";\n } else {\n insertTag = \"interpreter-after\";\n }\n } else {\n if (tag.afterCount > 0) {\n insertTag = \"interpreter-before\";\n } else {\n insertTag = \"interpreter\";\n }\n }\n\n if (snippet.contextBefore.isNotEmpty) buffer.writeln();\n buffer.write(\"<$insertTag>\");\n\n var code = formatCode(snippet.file.language, snippet.added, Format.print);\n // Discard the trailing newline so we don't end up with a blank paragraph\n // in InDesign.\n code = code.trimTrailingNewline();\n\n // Replace newlines with soft breaks so that InDesign treats the entire\n // snippet as a single paragraph and keeps it together.\n code = code.replaceAll(\"\\n\", \"
\");\n\n buffer.write(code);\n buffer.write(\"\");\n }\n\n if (snippet.contextAfter.isNotEmpty) {\n buffer.writeln();\n _writeContextXml(buffer, snippet.contextAfter, \"after\");\n }\n\n return buffer.toString();\n}\n\nvoid _writeContextHtml(Format format, StringBuffer buffer, List lines,\n {String cssClass}) {\n buffer.write(\"\");\n\n // The HTML spec mandates that a leading newline after '
' is ignored.\n  // https://html.spec.whatwg.org/#element-restrictions\n  // Some snippets deliberately start with a newline which needs to be\n  // preserved, so output an extra (discarded) newline in that case.\n  if (format.isWeb && lines.first.isEmpty) buffer.writeln();\n\n  for (var line in lines) {\n    buffer.writeln(line.escapeHtml);\n  }\n\n  buffer.write(\"
\");\n}\n\nvoid _writeContextXml(StringBuffer buffer, List lines, String tag) {\n if (lines.isEmpty) return;\n\n buffer.write(\"\");\n var first = true;\n for (var line in lines) {\n // Soft break, so that the context stays one paragraph.\n if (!first) buffer.write(\"
\");\n first = false;\n buffer.write(line.escapeHtml);\n }\n buffer.write(\"\");\n}\n" }, { "path": "tool/lib/src/markdown/html_renderer.dart", "content": "import 'package:markdown/markdown.dart';\n\n/// Custom Markdown to HTML renderer with some tweaks for the output we want.\nclass HtmlRenderer implements NodeVisitor {\n static const _blockTags = {\n \"blockquote\",\n \"div\",\n \"h1\",\n \"h2\",\n \"h3\",\n \"h4\",\n \"h5\",\n \"h6\",\n \"hr\",\n \"li\",\n \"ol\",\n \"p\",\n \"pre\",\n \"ul\",\n };\n\n StringBuffer buffer;\n\n final _elementStack = [];\n String _lastVisitedTag;\n\n String render(List nodes) {\n buffer = StringBuffer();\n\n for (final node in nodes) {\n node.accept(this);\n }\n\n buffer.writeln();\n\n return buffer.toString();\n }\n\n void visitText(Text text) {\n var content = text.text;\n\n // Put a newline before inline HTML markup for block-level tags.\n if (content.startsWith(\".\n buffer.write(' />');\n if (element.tag == 'br') buffer.write('\\n');\n return false;\n } else {\n _elementStack.add(element);\n buffer.write('>');\n return true;\n }\n }\n\n void visitElementAfter(Element element) {\n assert(identical(_elementStack.last, element));\n\n if (element.children != null &&\n element.children.isNotEmpty &&\n _blockTags.contains(_lastVisitedTag) &&\n _blockTags.contains(element.tag)) {\n buffer.writeln();\n } else if (element.tag == 'blockquote') {\n buffer.writeln();\n }\n buffer.write('');\n\n _lastVisitedTag = _elementStack.removeLast().tag;\n }\n}\n" }, { "path": "tool/lib/src/markdown/inline_syntax.dart", "content": "import 'package:charcode/ascii.dart';\nimport 'package:markdown/markdown.dart';\n\nimport '../format.dart';\n\nclass EllipseSyntax extends InlineSyntax {\n final Format _format;\n\n EllipseSyntax(this._format) : super(r\"\\.\\.\\. ?\", startCharacter: $dot);\n\n bool onMatch(InlineParser parser, Match match) {\n // A Unicode ellipsis doesn't have as much space between the dots as\n // Chicago style mandates so do our own thing.\n parser.addNode(Text(_format.isPrint\n ? \" . . . \"\n : ' . . . '));\n return true;\n }\n}\n\nclass ApostropheSyntax extends InlineSyntax {\n final Format _format;\n\n ApostropheSyntax(this._format) : super(r\"'\", startCharacter: $apostrophe);\n\n bool onMatch(InlineParser parser, Match match) {\n var before = -1;\n if (parser.pos > 0) {\n before = parser.charAt(parser.pos - 1);\n }\n var after = -1;\n if (parser.pos < parser.source.length - 1) {\n after = parser.charAt(parser.pos + 1);\n }\n\n var isRight = _isRight(before, after);\n String quote;\n if (_format.isPrint) {\n quote = isRight ? \"#8217\" : \"#8216\";\n } else {\n quote = isRight ? \"rsquo\" : \"lsquo\";\n }\n parser.addNode(Text(\"&$quote;\"));\n return true;\n }\n\n bool _isRight(int before, int after) {\n // Years like \"the '60s\".\n if (before == $space && after >= $0 && after <= $9) return true;\n\n // Possessive after code.\n if (before == $backquote && after == $s) return true;\n\n if (before == $space) return false;\n if (before == $lf) return false;\n\n // Default to right.\n return true;\n }\n}\n\nclass SmartQuoteSyntax extends InlineSyntax {\n final Format _format;\n\n SmartQuoteSyntax(this._format) : super(r'\"', startCharacter: $double_quote);\n\n bool onMatch(InlineParser parser, Match match) {\n var before = -1;\n if (parser.pos > 0) {\n before = parser.charAt(parser.pos - 1);\n }\n var after = -1;\n if (parser.pos < parser.source.length - 1) {\n after = parser.charAt(parser.pos + 1);\n }\n\n var isRight = _isRight(before, after);\n String quote;\n if (_format.isPrint) {\n quote = isRight ? \"#8221\" : \"#8220\";\n } else {\n quote = isRight ? \"rdquo\" : \"ldquo\";\n }\n\n parser.addNode(Text(\"&$quote;\"));\n return true;\n }\n\n bool _isRight(int before, int after) {\n if (after == $space) return true;\n if (before >= $a && before <= $z) return true;\n if (before >= $A && before <= $Z) return true;\n if (before >= $0 && before <= $9) return true;\n if (before == $dot) return true;\n if (before == $question) return true;\n if (before == $exclamation) return true;\n\n if (after == $colon) return true;\n if (after == $comma) return true;\n if (after == $dot) return true;\n\n // Default to left.\n return false;\n }\n}\n\nclass EmDashSyntax extends InlineSyntax {\n final Format _format;\n\n EmDashSyntax(this._format) : super(r\"\\s--\\s\");\n\n bool onMatch(InlineParser parser, Match match) {\n parser.addNode(\n Text(_format.isPrint ? '—' : '—'));\n return true;\n }\n}\n\n/// Remove newlines in paragraphs and turn them into spaces since InDesign\n/// treats them as line breaks.\nclass NewlineSyntax extends InlineSyntax {\n NewlineSyntax() : super(\"\\n\", startCharacter: $lf);\n\n bool onMatch(InlineParser parser, Match match) {\n parser.addNode(Text(\" \"));\n return true;\n }\n}\n" }, { "path": "tool/lib/src/markdown/markdown.dart", "content": "import 'package:markdown/markdown.dart' hide HtmlRenderer;\n\nimport '../book.dart';\nimport '../format.dart';\nimport '../page.dart';\nimport 'block_syntax.dart';\nimport 'code_syntax.dart';\nimport 'html_renderer.dart';\nimport 'inline_syntax.dart';\nimport 'xml_renderer.dart';\n\nString renderMarkdown(Book book, Page page, List lines, Format format) {\n var document = Document(blockSyntaxes: [\n BookHeaderSyntax(page, format),\n CodeTagBlockSyntax(book, page, format),\n HighlightedCodeBlockSyntax(format),\n ], inlineSyntaxes: [\n // Put inline Markdown code syntax before our smart quotes so that\n // quotes inside `code` spans don't get smartened.\n CodeSyntax(),\n EllipseSyntax(format),\n ApostropheSyntax(format),\n SmartQuoteSyntax(format),\n EmDashSyntax(format),\n if (format.isPrint) NewlineSyntax(),\n ], extensionSet: ExtensionSet.gitHubFlavored);\n\n var ast = document.parseLines(lines);\n if (format.isPrint) {\n return XmlRenderer().render(ast);\n } else {\n return HtmlRenderer().render(ast);\n }\n}\n" }, { "path": "tool/lib/src/markdown/xml_renderer.dart", "content": "import 'package:markdown/markdown.dart';\n\nfinal _imagePathPattern = RegExp(r'\"([^\"]+.png)\"');\n\n/// Matches opening XML tag names.\nfinal _tagPattern = RegExp(r\"<([a-z-_0-9]+)\");\n\nfinal _spanPattern = RegExp(r'');\n\nfinal _smallCapsPattern = RegExp(r'([A-Z]+)');\n\nclass XmlRenderer implements NodeVisitor {\n /// While building, also fill a StringBuffer with the minimal set of\n /// paragraphs needed to cover all tags in the book.\n static final tagFileBuffer = StringBuffer();\n\n /// Keeps track of which XML tags [tagFileBuffer] contains.\n static final allTags = {};\n\n /// The list of paragraph-level tags.\n final List<_Paragraph> _paragraphs = [];\n\n /// Whether we need to create a new paragraph before appending the next text.\n bool _pendingParagraph = true;\n\n /// The nested stack of current inline tags.\n final List<_Inline> _inlineStack = [];\n\n /// The stack tracking where we are in the document.\n _Context _context = _Context(\"main\");\n\n String render(List nodes) {\n for (final node in nodes) {\n node.accept(this);\n }\n\n var buffer = StringBuffer();\n buffer.writeln(\"\");\n\n _Paragraph previousMain;\n _Paragraph previousAside;\n\n for (var paragraph in _paragraphs) {\n String text;\n\n if (paragraph.context.has(\"aside\")) {\n text = paragraph.prettyPrint(previousAside);\n previousAside = paragraph;\n } else {\n text = paragraph.prettyPrint(previousMain);\n previousMain = paragraph;\n\n // Reached the end of an aside.\n previousAside = null;\n }\n\n buffer.write(text);\n\n // Only add the paragraph to the tag file buffer if it has a unique tag.\n var tags = _tagPattern.allMatches(text).map((match) => match[1]).toSet();\n if (tags.difference(allTags).isNotEmpty) {\n tagFileBuffer.write(text);\n allTags.addAll(tags);\n }\n }\n\n buffer.writeln(\"\");\n return buffer.toString();\n }\n\n void visitText(Text node) {\n var text = node.text;\n\n if (text.isEmpty) return;\n\n // There are a couple of hand-coded HTML ellipses inside an HTML table.\n text = text.replaceAll(\n ' . . .', \"…\");\n\n // Convert the small-caps bitwise operator spans in \"Optimization\" to\n // custom tags.\n text = text.replaceAllMapped(\n _smallCapsPattern, (match) => \"${match[1]}\");\n\n text = text\n .replaceAll(\"é\", \"é\")\n .replaceAll(\" \", \" \")\n .replaceAll(\"“\", \"“\")\n .replaceAll(\" \", \" \")\n .replaceAll(\"”\", \"”\")\n .replaceAll(\"’\", \"’\")\n .replaceAll(\"→\", \"→\")\n .replaceAll(\"§\", \"§\")\n .replaceAll(\" \", \" \")\n .replaceAll(\"×\", \"×\")\n .replaceAll(\"
\", \"
\");\n\n // Don't send tables to InDesign as XML.\n text = text\n .replaceAll(\"\", \"[table]\")\n .replaceAll(\"
\", \"[/table]\")\n .replaceAll(\"\", \"[thead]\")\n .replaceAll(\"\", \"[/thead]\")\n .replaceAll(\"\", \"[tbody]\")\n .replaceAll(\"\", \"[/tbody]\")\n .replaceAll(\"\", \"[tr]\")\n .replaceAll(\"\", \"[/tr]\")\n .replaceAll(\"\", \"[td]\")\n .replaceAll(\"\", \"[/td]\");\n\n // Turn aside span locators into little visible markers.\n text = text\n .replaceAll(_spanPattern, \"@\")\n .replaceAll(\"\", \"\");\n\n // Discard the challenge and design note divs.\n if (text.startsWith(\"\")) return;\n\n // Convert image tags to just their paths.\n if (text.startsWith(\"\") ||\n // \"Representing Code\" has a few inserted snippets with no location tag.\n text.startsWith(\"\") ||\n text.startsWith(\"\") ||\n text.startsWith(\"\")) {\n _push(\"xml\");\n _addText(text);\n _pop();\n return;\n }\n\n // Since aside tags appear in the Markdown as literal HTML, they are parsed\n // as text, not Markdown elements.\n if (text.startsWith(\"\")) {\n _pop();\n return;\n }\n\n if (text.trimLeft().startsWith(\"\")) {\n _push(\"xml\");\n\n // Use a custom inline style for cite emphasis.\n text = text\n .replaceAll(\"\", \"\")\n .replaceAll(\"\", \"\");\n\n _addText(text.trimLeft());\n } else if (_inlineStack.isNotEmpty) {\n // We're in an inline tag, so add it to that.\n _inlineStack.last.text += text;\n } else {\n if (_context.name == \"xml\") {\n // Hackish. Assume the only tags inside XML blocks are in cites.\n text = text\n .replaceAll(\"\", \"\")\n .replaceAll(\"\", \"\");\n }\n\n _addText(text);\n }\n\n if (text.endsWith(\"\")) _pop();\n }\n\n bool visitElementBefore(Element element) {\n switch (element.tag) {\n case \"p\":\n _resetParagraph();\n break;\n\n case \"blockquote\":\n _push(\"quote\");\n break;\n\n case \"h2\":\n var text = element.textContent;\n if (text == \"Challenges\") {\n _context = _Context(\"challenges\");\n } else if (text.contains(\"Design Note\")) {\n _context = _Context(\"design\");\n }\n _push(\"heading\");\n break;\n\n case \"h3\":\n _push(\"subheading\");\n break;\n\n case \"ol\":\n _push(\"ordered\");\n\n // Immediately push a subcontext to mark the first list item.\n _push(\"first\");\n break;\n\n case \"pre\":\n _push(\"pre\");\n break;\n\n case \"ul\":\n _push(\"unordered\");\n break;\n\n case \"li\":\n // If we're on the first item, discard it and replace it with the next\n // item. The first item restarts numbering but later ones don't.\n if (_context.name != \"first\") _push(\"item\");\n break;\n\n case \"a\":\n // TODO: What do we want to do with links? Highlight them somehow so\n // that I decide if the surrounding text needs tweaking?\n break;\n\n case \"code\":\n case \"em\":\n case \"small\":\n case \"strong\":\n // Inline tags.\n\n // If we're in an inline tags already, flatten them by emitting inline\n // segments for any text they have. Leave them on the stack so that\n // they get resumed when the nested inline tags end.\n var tagParts = [element.tag];\n for (var i = 0; i < _inlineStack.length; i++) {\n var inline = _inlineStack[i];\n if (inline.text.isNotEmpty) {\n _addInline(inline);\n _inlineStack[i] = _Inline(inline.tag);\n }\n\n tagParts.add(inline.tag);\n }\n\n String tag;\n if (tagParts.contains(\"code\")) {\n // Code formatting wipes out italics or bold.\n tag = \"code\";\n } else {\n tagParts.sort();\n tag = tagParts.join(\"-\");\n }\n // Make a tag name that includes all nested tags. We'll define separate\n // styles for each combination.\n _inlineStack.add(_Inline(tag));\n break;\n\n default:\n print(\"Unexpected open tag ${element.tag}.\");\n }\n\n return !element.isEmpty;\n }\n\n void visitElementAfter(Element element) {\n switch (element.tag) {\n case \"blockquote\":\n case \"h2\":\n case \"h3\":\n case \"pre\":\n _pop();\n break;\n\n case \"ol\":\n case \"ul\":\n // If we still have a context for the item, it means we have a Markdown\n // list with no paragraph tags inside the items. There are a couple of\n // those in the book.\n if (_context.name == \"first\" || _context.name == \"item\") _pop();\n\n // Pop the list itself.\n _pop();\n break;\n\n case \"a\":\n // Nothing to do.\n break;\n\n case \"li\":\n case \"p\":\n // The first paragraph in each list item has a special style so that\n // apply the bullet or number. Later paragraphs in the same list item\n // do not.\n\n // We match both

and

  • so that lists without paragraphs inside\n // don't leave lingering item contexts.\n if (_context.name == \"first\" || _context.name == \"item\") _pop();\n break;\n\n case \"code\":\n case \"em\":\n case \"small\":\n case \"strong\":\n // Inline tags.\n _addInline(_inlineStack.removeLast());\n break;\n\n default:\n print(\"Unexpected close tag ${element.tag}.\");\n }\n }\n\n void _push(String name) {\n _context = _Context(name, _context);\n _resetParagraph();\n }\n\n void _pop() {\n _context = _context.parent;\n _resetParagraph();\n }\n\n void _addText(String text) {\n _flushParagraph();\n\n // Discard any leading whitespace at the beginning of list items.\n var paragraph = _paragraphs.last;\n if (paragraph.contents.isEmpty &&\n (_context.has(\"ordered\") || _context.has(\"unordered\"))) {\n text = text.trimLeft();\n }\n\n paragraph.contents.add(_Inline(null, text));\n }\n\n void _addInline(_Inline inline) {\n _flushParagraph();\n _paragraphs.last.contents.add(inline);\n }\n\n void _resetParagraph() {\n _pendingParagraph = true;\n }\n\n void _flushParagraph() {\n if (!_pendingParagraph) return;\n _paragraphs.add(_Paragraph(_context));\n _pendingParagraph = false;\n }\n}\n\nclass _Context {\n final String name;\n final _Context parent;\n\n _Context(this.name, [this.parent]);\n\n /// Whether any of the contexts in this chain are [name].\n bool has(String name) {\n var context = this;\n while (context != null) {\n if (context.name == name) return true;\n context = context.parent;\n }\n\n return false;\n }\n\n /// Whether [parent] has [name].\n bool isIn(String name) => parent != null && parent.has(name);\n\n /// How many levels of list nesting this context contains.\n int get listDepth {\n var depth = 0;\n\n for (var context = this; context != null; context = context.parent) {\n if (context.name == \"ordered\" || context.name == \"unordered\") {\n depth++;\n } else if (context.name == \"aside\") {\n // Content inside an aside inside a list item isn't really part of the\n // list.\n break;\n }\n }\n\n return depth;\n }\n\n String get paragraphTag {\n var tag = name;\n var depth = listDepth;\n if (depth > 2) print(\"Unexpected deep list nesting $this.\");\n\n switch (tag) {\n case \"main\":\n return \"p\";\n case \"main\":\n return \"p\";\n case \"challenges\":\n // There's only paragraph of non-list prose text and that's also\n // indented like a list (so that it lines up with the heading), so just\n // use the same style for both.\n return \"challenges-list-p\";\n case \"design\":\n return \"design-p\";\n case \"aside\":\n return \"aside\";\n case \"xml\":\n return \"xml\";\n\n case \"first\":\n case \"item\":\n tag = \"${parent.name}-$tag\";\n if (depth > 1) tag = \"sublist-$tag\";\n break;\n\n case \"ordered\":\n case \"unordered\":\n tag = \"list-p\";\n if (depth > 1) tag = \"sublist-$tag\";\n break;\n\n default:\n if (depth > 1) {\n tag = \"sublist-$tag\";\n } else if (depth > 0) {\n tag = \"list-$tag\";\n }\n }\n\n if (isIn(\"aside\")) {\n tag = \"aside-$tag\";\n } else if (isIn(\"challenges\")) {\n tag = \"challenges-$tag\";\n } else if (isIn(\"design\")) {\n tag = \"design-$tag\";\n }\n\n return tag;\n }\n\n /// The prefix to apply to inline tags within this context or the empty string\n /// it none should be added.\n String get inlinePrefix {\n if (has(\"aside\")) return \"aside\";\n if (has(\"challenges\")) return \"challenges\";\n if (has(\"design\")) return \"design\";\n if (has(\"quote\")) return \"quote\";\n\n return \"\";\n }\n\n String toString() {\n if (parent == null) return name;\n return \"$parent > $name\";\n }\n}\n\n/// A paragraph-level tag that contains text and inline tags.\nclass _Paragraph {\n final _Context context;\n\n final List<_Inline> contents = [];\n\n _Paragraph(this.context);\n\n bool _isNext(String tag, String previousTag) {\n const nextTags = {\n \"aside\",\n \"challenges-p\",\n \"challenges-list-p\",\n \"design-p\",\n \"design-list-p\",\n \"list-p\",\n \"p\"\n };\n\n if (tag == previousTag) return nextTags.contains(tag);\n\n // The paragraph after a bullet item is also a next.\n if (tag.endsWith(\"list-p\")) {\n // This includes both \"unordered\" and \"ordered\", tags that start with\n // \"challenges\" or \"design\", and ones that end with \"first\" or \"item\".\n return previousTag.contains(\"ordered-\");\n }\n\n return false;\n }\n\n String prettyPrint(_Paragraph previous) {\n var buffer = StringBuffer();\n var tag = context.paragraphTag;\n\n if (previous != null && _isNext(tag, previous.context.paragraphTag)) {\n tag += \"-next\";\n }\n\n if (tag != \"xml\") buffer.write(\"<$tag>\");\n\n for (var inline in contents) {\n inline.prettyPrint(buffer, context);\n }\n\n if (tag != \"xml\") buffer.write(\"\");\n buffer.writeln();\n return buffer.toString();\n }\n}\n\n/// An inline tag or plain text.\nclass _Inline {\n /// The tag name if this is an inline tag or `null` if it is text.\n final String tag;\n\n String text;\n\n _Inline(this.tag, [this.text = \"\"]);\n\n bool get isText => tag == null;\n\n void prettyPrint(StringBuffer buffer, _Context context) {\n if (tag == null) {\n buffer.write(text);\n return;\n }\n\n var fullTag = tag;\n var prefix = context.inlinePrefix;\n if (prefix != \"\") fullTag = \"$prefix-$fullTag\";\n\n buffer.write(\"<$fullTag>$text\");\n }\n}\n" }, { "path": "tool/lib/src/mustache.dart", "content": "/// Creates the data map and renders the Mustache templates to HTML.\nimport 'dart:io';\n\nimport 'package:mustache_template/mustache_template.dart';\nimport 'package:path/path.dart' as p;\n\nimport 'book.dart';\nimport 'page.dart';\nimport 'text.dart';\n\n/// Maintains the cache of loaded partials and allows rendering templates.\nclass Mustache {\n /// The directory where template files can be found.\n final String _templateDirectory;\n\n final Map _templates = {};\n\n Mustache([String templateDirectory])\n : _templateDirectory = templateDirectory ?? p.join(\"asset\", \"mustache\");\n\n String render(Book book, Page page, String body, {String template}) {\n var part = page.part?.title;\n\n var up = \"Table of Contents\";\n if (part != null) {\n up = part;\n } else if (page.title == \"Table of Contents\") {\n up = \"Crafting Interpreters\";\n }\n\n var previousPage = book.adjacentPage(page, -1);\n var nextPage = book.adjacentPage(page, 1);\n String nextType;\n if (nextPage != null && nextPage.isChapter) {\n nextType = \"Chapter\";\n } else if (nextPage != null && nextPage.isPart) {\n nextType = \"Part\";\n }\n\n List> chapters;\n if (page.isPart) {\n chapters = _makeChapterList(page);\n }\n\n var isFrontmatter = const {\n \"Acknowledgements\",\n \"Dedication\",\n }.contains(page.title);\n\n var data = {\n \"is_chapter\": part != null,\n \"is_part\": part == null && page.title != null && !isFrontmatter,\n \"is_frontmatter\": isFrontmatter,\n \"title\": page.title,\n \"part\": part,\n \"body\": body,\n \"sections\": _makeSections(page),\n \"chapters\": chapters,\n \"design_note\": page.designNote,\n \"has_design_note\": page.designNote != null,\n \"has_challenges\": page.hasChallenges,\n \"has_challenges_or_design_note\":\n page.hasChallenges || page.designNote != null,\n \"has_number\": page.numberString != \"\",\n \"number\": page.numberString,\n // Previous page.\n \"has_prev\": previousPage != null,\n \"prev\": previousPage?.title,\n \"prev_file\": previousPage?.fileName,\n // Next page.\n \"has_next\": nextPage != null,\n \"next\": nextPage?.title,\n \"next_file\": nextPage?.fileName,\n \"next_type\": nextType,\n \"has_up\": up != null,\n \"up\": up,\n \"up_file\": up != null ? toFileName(up) : null,\n // TODO: Only need this for contents page.\n \"part_1\": _makePartData(book, 0),\n \"part_2\": _makePartData(book, 1),\n \"part_3\": _makePartData(book, 2),\n };\n\n return _load(template ?? page.template).renderString(data);\n }\n\n Map _makePartData(Book book, int partIndex) {\n var partPage = book.parts[partIndex];\n return {\n \"title\": partPage.title,\n \"number\": partPage.numberString,\n \"file\": partPage.fileName,\n \"chapters\": _makeChapterList(partPage)\n };\n }\n\n List> _makeChapterList(Page part) {\n return [\n for (var chapter in part.chapters)\n {\n \"title\": chapter.title,\n \"number\": chapter.numberString,\n \"file\": chapter.fileName,\n \"design_note\": chapter.designNote?.replaceAll(\"'\", \"’\"),\n }\n ];\n }\n\n List> _makeSections(Page page) {\n var sections = >[];\n\n for (var header in page.headers.values) {\n if (!header.isSpecial && header.level == 2) {\n sections.add({\n \"name\": header.name,\n \"anchor\": header.anchor,\n \"index\": header.headerIndex\n });\n }\n }\n\n return sections;\n }\n\n Template _load(String name) {\n return _templates.putIfAbsent(name, () {\n var path = p.join(_templateDirectory, \"$name.html\");\n return Template(File(path).readAsStringSync(),\n name: path, partialResolver: _load);\n });\n }\n}\n" }, { "path": "tool/lib/src/page.dart", "content": "import 'package:path/path.dart' as p;\n\nimport 'code_tag.dart';\nimport 'page_parser.dart';\nimport 'text.dart';\n\n/// One page (in the HTML sense) of the book.\n///\n/// Each chapter, part introduction, and backmatter section is a page.\nclass Page {\n /// The title of this page.\n final String title;\n\n /// The chapter or part number, like \"12\", \"II\", or \"\".\n final String numberString;\n\n /// The numeric index of the page in chapter order.\n ///\n /// Used to determine which order snippets appear in the book.\n final int ordinal;\n\n /// If this page is a part page, the list of chapter pages it contains.\n final List chapters = [];\n\n /// If this page is a chapter page, the part that contains this page.\n final Page part;\n\n PageFile _file;\n\n Page(this.title, this.part, this.numberString, this.ordinal);\n\n /// The base file path and URI for the page, without any extension.\n String get fileName => toFileName(title);\n\n /// The path to this page's Markdown source file.\n String get markdownPath => p.join(\"book\", \"$fileName.md\");\n\n /// The path to this page's generated HTML file.\n String get htmlPath => p.join(\"site\", \"$fileName.html\");\n\n /// Whether this page is a chapter page, as opposed to a part.\n bool get isChapter => part != null;\n\n /// Whether this page is a part page, as opposed to a chapter.\n bool get isPart => part == null;\n\n /// The code language used for this chapter page or `null` if this isn't one\n /// of the main chapter pages.\n String get language {\n if (isPart) return null;\n if (part.title == \"A Tree-Walk Interpreter\") return \"java\";\n if (part.title == \"A Bytecode Virtual Machine\") return \"c\";\n return null;\n }\n\n String get shortName {\n var number = numberString.padLeft(2, \"0\");\n\n var words = title.split(\" \");\n var word = words.first.toLowerCase();\n if (word == \"a\" || word == \"the\") word = words[1].toLowerCase();\n\n return \"chap${number}_$word\";\n }\n\n List get lines => _ensureFile().lines;\n\n String get template {\n if (title == \"Crafting Interpreters\") return \"index\";\n if (title == \"Table of Contents\") return \"contents\";\n return \"page\";\n }\n\n Map get headers => _ensureFile().headers;\n\n bool get hasChallenges => _ensureFile().hasChallenges;\n\n String get designNote => _ensureFile().designNote;\n\n Iterable get codeTags => _ensureFile().codeTags.values;\n\n CodeTag findCodeTag(String name) {\n // Return fake tags for the placeholders.\n if (name == \"omit\") return CodeTag(this, \"omit\", 9998, 0, 0, false);\n if (name == \"not-yet\") return CodeTag(this, \"omit\", 9999, 0, 0, false);\n\n var codeTag = _ensureFile().codeTags[name];\n if (codeTag != null) return codeTag;\n\n throw ArgumentError(\"Could not find code tag '$name'.\");\n }\n\n String toString() => title;\n\n /// Lazily parse the Markdown file for the page.\n PageFile _ensureFile() => _file ??= parsePage(this);\n}\n\n/// The data for a page parsed from the Markdown source.\nclass PageFile {\n final List lines;\n final Map headers;\n final bool hasChallenges;\n\n /// The name of the design note in this page, or `null` if there is none.\n final String designNote;\n\n final Map codeTags;\n\n PageFile(this.lines, this.headers, this.hasChallenges, this.designNote,\n this.codeTags);\n}\n\n/// A section header in a page.\nclass Header {\n /// The header depth: 1 is the page title, 2 header, 3 subheader.\n final int level;\n final int headerIndex;\n final int subheaderIndex;\n final String name;\n\n Header(this.level, this.headerIndex, this.subheaderIndex, this.name);\n\n /// Whether this header is for the special \"Challenges\" or \"Design Note\"\n /// sections.\n bool get isSpecial => isChallenges || isDesignNote;\n\n bool get isChallenges {\n // Check for a subheader because there is a \"Challenges\" *subheader* in\n // the Introduction.\n return name == \"Challenges\" && level == 2;\n }\n\n bool get isDesignNote => name.startsWith(\"Design Note:\");\n\n String get anchor {\n if (isChallenges) return \"challenges\";\n if (isDesignNote) return \"design-note\";\n return toFileName(name);\n }\n}\n" }, { "path": "tool/lib/src/page_parser.dart", "content": "import 'dart:io';\n\nimport 'code_tag.dart';\nimport 'page.dart';\nimport 'text.dart';\n\nfinal _codePattern = RegExp(r\"^\\^code ([-a-z0-9]+)( \\(([^)]+)\\))?$\");\nfinal _headerPattern = RegExp(r\"^(#{1,3}) \");\nfinal _beforePattern = RegExp(r\"(\\d+) before\");\nfinal _afterPattern = RegExp(r\"(\\d+) after\");\n\n/// Parses the contents of the Markdown file for [page] to extract its metadata,\n/// code tags, section headers, etc.\nPageFile parsePage(Page page) {\n var headers = {};\n var codeTagsByName = {};\n String designNote;\n var hasChallenges = false;\n\n var headerIndex = 0;\n var subheaderIndex = 0;\n\n var lines = File(page.markdownPath).readAsLinesSync();\n for (var i = 0; i < lines.length; i++) {\n var line = lines[i];\n\n var match = _codePattern.firstMatch(line);\n if (match != null) {\n var codeTag =\n _createCodeTag(page, codeTagsByName.length, match[1], match[3]);\n codeTagsByName[codeTag.name] = codeTag;\n continue;\n }\n\n match = _headerPattern.firstMatch(line);\n if (match != null) {\n // Keep track of the headers so we can add section navigation for them.\n var headerType = match[1];\n var level = headerType.length;\n var name = line.substring(level).trim().pretty;\n\n if (level == 2) {\n headerIndex += 1;\n subheaderIndex = 0;\n } else if (level == 3) {\n subheaderIndex += 1;\n }\n\n var header =\n Header(level, headerIndex, level == 3 ? subheaderIndex : null, name);\n\n if (header.isChallenges) hasChallenges = true;\n if (header.isDesignNote) {\n designNote = header.name.substring(\"Design Note: \".length);\n }\n\n headers[line] = header;\n }\n }\n\n// # Validate that every snippet for the chapter is included.\n// for name, snippet in snippets.items():\n// if name != 'not-yet' and name != 'omit' and snippet != False:\n// errors.append(\"Unused snippet {}\".format(name))\n//\n// # Show any errors at the top of the file.\n// if errors:\n// error_markdown = \"\"\n// for error in errors:\n// error_markdown += \"**Error: {}**\\n\\n\".format(error)\n// contents = error_markdown + contents\n//\n return PageFile(lines, headers, hasChallenges, designNote, codeTagsByName);\n}\n\nCodeTag _createCodeTag(Page page, int index, String name, String options) {\n // Parse the location annotations after the name, if present.\n var showLocation = true;\n var beforeCount = 0;\n var afterCount = 0;\n\n if (options != null) {\n for (var option in options.split(\", \")) {\n if (option == \"no location\") {\n showLocation = false;\n continue;\n }\n\n var match = _beforePattern.firstMatch(option);\n if (match != null) {\n beforeCount = int.parse(match[1]);\n continue;\n }\n\n match = _afterPattern.firstMatch(option);\n if (match != null) {\n afterCount = int.parse(match[1]);\n continue;\n }\n\n throw \"Unknown code option '$option'\";\n }\n }\n\n return CodeTag(page, name, index, beforeCount, afterCount, showLocation);\n}\n" }, { "path": "tool/lib/src/snippet.dart", "content": "import 'book.dart';\nimport 'code_tag.dart';\nimport 'location.dart';\nimport 'text.dart';\n\n/// A snippet of source code that is inserted in the book.\nclass Snippet {\n final SourceFile file;\n final CodeTag tag;\n\n Location _location;\n\n int _firstLine;\n int _lastLine;\n\n Location get precedingLocation => _precedingLocation;\n Location _precedingLocation;\n\n /// If the snippet replaces a line with the same line but with a trailing\n /// comma, this is that line (with the comma).\n String get addedComma => _addedComma;\n String _addedComma;\n\n final List added = [];\n final List removed = [];\n\n final List contextBefore = [];\n final List contextAfter = [];\n\n Snippet(this.file, this.tag);\n\n void addLine(int lineIndex, SourceLine line) {\n if (added.isEmpty) {\n _location = line.location;\n _firstLine = lineIndex;\n }\n added.add(line.text);\n\n // Assume that we add the removed lines in order.\n _lastLine = lineIndex;\n }\n\n void removeLine(int lineIndex, SourceLine line) {\n removed.add(line.text);\n\n // Assume that we add the removed lines in order.\n _lastLine = lineIndex;\n }\n\n /// Describes where in the file this snippet appears. Returns a list of HTML\n /// strings.\n List get locationHtmlLines {\n var result = [\"${file.nicePath}\"];\n\n var html = _location.toHtml(precedingLocation, removed);\n if (html != null) result.add(html);\n\n if (removed.isNotEmpty && added.isNotEmpty) {\n result.add(\"replace ${removed.length} line${pluralize(removed)}\");\n } else if (removed.isNotEmpty && added.isEmpty) {\n result.add(\"remove ${removed.length} line${pluralize(removed)}\");\n }\n\n if (addedComma != null) {\n result.add(\"add “,” to previous line\");\n }\n\n return result;\n }\n\n /// Describes where in the file this snippet appears.\n String get locationXml {\n var result = StringBuffer();\n result.write(\"${file.nicePath}\");\n\n var xml = _location.toXml(precedingLocation, removed);\n var changes = [\n if (xml != null) xml,\n if (removed.isNotEmpty && added.isNotEmpty)\n \"replace ${removed.length} line${pluralize(removed)}\"\n else if (removed.isNotEmpty && added.isEmpty)\n \"remove ${removed.length} line${pluralize(removed)}\",\n if (addedComma != null)\n \"add “,” to previous line\"\n ].map((change) => \"$change\");\n\n if (changes.isNotEmpty) {\n result.writeln();\n result.writeAll(changes, \"\\n\");\n }\n\n return result.toString();\n }\n\n String toString() => \"${file.nicePath} ${tag.name}\";\n\n /// Calculate the surrounding context information for this snippet.\n void calculateContext() {\n // Get the preceding lines.\n for (var i = _firstLine - 1;\n i >= 0 && contextBefore.length < tag.beforeCount;\n i--) {\n var line = file.lines[i];\n if (!line.isPresent(tag)) continue;\n contextBefore.insert(0, line.text);\n }\n\n // Get the following lines.\n for (var i = _lastLine + 1;\n i < file.lines.length && contextAfter.length < tag.afterCount;\n i++) {\n var line = file.lines[i];\n if (line.isPresent(tag)) contextAfter.add(line.text);\n }\n\n // Get the preceding location.\n // TODO: This constant is somewhat arbitrary. Come up with a more precise\n // way to track the preceding location.\n int checkedLines = 0;\n for (var i = _firstLine - 1; i >= 0 && checkedLines <= 4; i--) {\n var line = file.lines[i];\n if (!line.isPresent(tag)) continue;\n checkedLines++;\n\n // Store the most precise preceding location we find.\n if (_precedingLocation == null ||\n line.location.depth > _precedingLocation.depth) {\n _precedingLocation = line.location;\n }\n }\n\n // Update the current location based on surrounding lines.\n var hasCodeBefore = contextBefore.isNotEmpty;\n var hasCodeAfter = contextAfter.isNotEmpty;\n for (var i = _firstLine - 1; !hasCodeBefore && i >= 0; i--) {\n hasCodeBefore = file.lines[i].isPresent(tag);\n }\n\n for (var i = _lastLine + 1; !hasCodeAfter && i < file.lines.length; i++) {\n hasCodeAfter = file.lines[i].isPresent(tag);\n }\n\n if (!hasCodeBefore) {\n _location = Location(null, hasCodeAfter ? \"top\" : \"new\", null);\n }\n\n // Find line changes that just add a trailing comma.\n if (added.isNotEmpty &&\n removed.isNotEmpty &&\n added.first == \"${removed.last},\") {\n _addedComma = added.first;\n added.removeAt(0);\n removed.removeLast();\n }\n }\n}\n" }, { "path": "tool/lib/src/source_file_parser.dart", "content": "import 'dart:io';\n\nimport 'book.dart';\nimport 'code_tag.dart';\nimport 'location.dart';\nimport 'page.dart';\n\nfinal _blockPattern = RegExp(\n r\"^/\\* ([A-Z][A-Za-z\\s]+) ([-a-z0-9]+) < ([A-Z][A-Za-z\\s]+) ([-a-z0-9]+)$\");\nfinal _blockSnippetPattern = RegExp(r\"^/\\* < ([-a-z0-9]+)$\");\nfinal _beginSnippetPattern = RegExp(r\"^//> ([-a-z0-9]+)$\");\nfinal _endSnippetPattern = RegExp(r\"^//< ([-a-z0-9]+)$\");\nfinal _beginChapterPattern = RegExp(r\"^//> ([A-Z][A-Za-z\\s]+) ([-a-z0-9]+)$\");\nfinal _endChapterPattern = RegExp(r\"^//< ([A-Z][A-Za-z\\s]+) ([-a-z0-9]+)$\");\n\n// Hacky regexes that matches various declarations.\nfinal _constructorPattern = RegExp(r\"^ ([A-Z][a-z]\\w+)\\(\");\nfinal _functionPattern = RegExp(r\"(\\w+)>*\\*? (\\w+)\\(([^)]*)\");\nfinal _variablePattern = RegExp(r\"^\\w+\\*? (\\w+)(;| = )\");\nfinal _structPattern = RegExp(r\"^struct (\\w+)? {$\");\nfinal _typePattern =\n RegExp(r\"(public )?(abstract )?(class|enum|interface) ([A-Z]\\w+)\");\nfinal _namedTypedefPattern = RegExp(r\"^typedef (enum|struct|union) (\\w+) {$\");\nfinal _unnamedTypedefPattern = RegExp(r\"^typedef (enum|struct|union) {$\");\nfinal _typedefNamePattern = RegExp(r\"^\\} (\\w+);$\");\n\n/// Reserved words that can appear like a return type in a function declaration\n/// but shouldn't be treated as one.\nconst _keywords = {\"new\", \"return\", \"throw\"};\n\nclass SourceFileParser {\n final Book _book;\n final SourceFile _file;\n final List _lines;\n final List<_ParseState> _states = [];\n\n Location _unnamedTypedef;\n\n Location _location;\n Location _locationBeforeBlock;\n\n SourceFileParser(this._book, String path, String relative)\n : _file = SourceFile(relative),\n _lines = File(path).readAsLinesSync() {\n _location = Location(null, \"file\", _file.nicePath);\n }\n\n SourceFile parse() {\n// line_num = 1\n// handled = False\n//\n// def error(message):\n// print(\"Error: {} line {}: {}\".format(relative, line_num, message),\n// file=sys.stderr)\n// source_code.errors[state.start.chapter].append(\n// \"{} line {}: {}\".format(relative, line_num, message))\n//\n // Split the source file into lines.\n// printed_file = False\n// line_num = 1\n for (var i = 0; i < _lines.length; i++) {\n var line = _lines[i].trimRight();\n// handled = False\n//\n// # Report any lines that are too long.\n// trimmed = re.sub(r'// \\[([-a-z0-9]+)\\]', '', line)\n// if len(trimmed) > 72 and not '/*' in trimmed:\n// if not printed_file:\n// print(\"Long line in {}:\".format(file.path))\n// printed_file = True\n// print(\"{0:4} ({1:2} chars): {2}\".format(line_num, len(trimmed), trimmed))\n//\n\n _updateLocationBefore(line, i);\n\n if (!_updateState(line)) {\n var sourceLine =\n SourceLine(line, _location, _currentState.start, _currentState.end);\n _file.lines.add(sourceLine);\n }\n\n _updateLocationAfter(line);\n//\n// line_num += 1\n }\n\n// # \".parent.parent\" because there is always the top \"null\" state.\n// if state.parent != None and state.parent.parent != None:\n// print(\"{}: Ended with more than one state on the stack.\".format(relative),\n// file=sys.stderr)\n// s = state\n// while s.parent != None:\n// print(\" {}\".format(s.start), file=sys.stderr)\n// s = s.parent\n// sys.exit(1)\n//\n\n // TODO: Validate that we don't define two snippets with the same chapter\n // and number. A snippet may end up in disjoint lines in the final output\n // because a later snippet is inserted in it, but it shouldn't be explicitly\n // authored that way.\n return _file;\n }\n\n /// Keep track of the current location where the parser is in the source file.\n void _updateLocationBefore(String line, int lineIndex) {\n // See if we reached a new function or method declaration.\n var match = _functionPattern.firstMatch(line);\n if (match != null &&\n !line.contains(\"#define\") &&\n !_keywords.contains(match[1])) {\n // Hack. Don't get caught by comments or string literals.\n if (!line.contains(\"//\") && !line.contains('\"')) {\n var isFunctionDeclaration = line.endsWith(\";\");\n\n // Hack: Handle multi-line declarations.\n if (line.endsWith(\",\") && _lines[lineIndex + 1].endsWith(\";\")) {\n isFunctionDeclaration = true;\n }\n\n _location = Location(_location,\n _file.language == \"java\" ? \"method\" : \"function\", match[2],\n signature: match[3], isFunctionDeclaration: isFunctionDeclaration);\n return;\n }\n }\n\n match = _constructorPattern.firstMatch(line);\n if (match != null) {\n _location = Location(_location, \"constructor\", match[1]);\n return;\n }\n\n match = _typePattern.firstMatch(line);\n if (match != null) {\n // Hack. Don't get caught by comments or string literals.\n if (!line.contains(\"//\") && !line.contains('\"')) {\n var kind = match[3];\n var name = match[4];\n _location = Location(_location, kind, name);\n }\n return;\n }\n\n match = _structPattern.firstMatch(line);\n if (match != null) {\n _location = Location(_location, \"struct\", match[1]);\n return;\n }\n\n match = _namedTypedefPattern.firstMatch(line);\n if (match != null) {\n _location = Location(_location, match[1], match[2]);\n return;\n }\n\n match = _unnamedTypedefPattern.firstMatch(line);\n if (match != null) {\n // We don't know the name of the typedef yet.\n _location = Location(_location, match[1], null);\n _unnamedTypedef = _location;\n return;\n }\n\n match = _variablePattern.firstMatch(line);\n if (match != null) {\n _location = Location(_location, \"variable\", match[1]);\n return;\n }\n }\n\n void _updateLocationAfter(String line) {\n var match = _typedefNamePattern.firstMatch(line);\n if (match != null) {\n // Now we know the typedef name.\n _unnamedTypedef?.name = match[1];\n _unnamedTypedef = null;\n _location = _location.parent;\n }\n\n // Use \"startsWith\" to include lines like \"} [aside-marker]\".\n if (line.startsWith(\"}\")) {\n _location = _location.popToDepth(0);\n } else if (line.startsWith(\" }\")) {\n _location = _location.popToDepth(1);\n } else if (line.startsWith(\" }\")) {\n _location = _location.popToDepth(2);\n }\n\n // If we reached a function declaration, not a definition, then it's done\n // after one line.\n if (_location.isFunctionDeclaration) {\n _location = _location.parent;\n }\n\n // Module variables are only a single line.\n if (_location.kind == \"variable\") {\n _location = _location.parent;\n }\n\n // Hack. There is a one-line class in Parser.java.\n if (line.contains(\"class ParseError\")) {\n _location = _location.parent;\n }\n }\n\n /// Processes any [line] that changes what snippet the parser is currently in.\n ///\n /// Returns `true` if the line contained a snippet annotation.\n bool _updateState(String line) {\n var match = _blockPattern.firstMatch(line);\n if (match != null) {\n _push(\n startChapter: _book.findChapter(match[1]),\n startName: match[2],\n endChapter: _book.findChapter(match[3]),\n endName: match[4]);\n _locationBeforeBlock = _location;\n return true;\n }\n\n match = _blockSnippetPattern.firstMatch(line);\n if (match != null) {\n _push(endChapter: _currentState.start.chapter, endName: match[1]);\n _locationBeforeBlock = _location;\n return true;\n }\n\n if (line.trim() == \"*/\" && _currentState.end != null) {\n _location = _locationBeforeBlock;\n _pop();\n return true;\n }\n\n match = _beginSnippetPattern.firstMatch(line);\n if (match != null) {\n var name = match[1];\n// var tag = source_code.find_snippet_tag(state.start.chapter, name);\n// if tag < state.start:\n// error(\"Can't push earlier snippet {} from {}.\".format(name, state.start.name))\n// elif tag == state.start:\n// error(\"Can't push to same snippet {}.\".format(name))\n _push(startName: name);\n return true;\n }\n\n match = _endSnippetPattern.firstMatch(line);\n if (match != null) {\n// var name = match[1];\n// if name != state.start.name:\n// error(\"Expecting to pop {} but got {}.\".format(state.start.name, name))\n// if state.parent.start.chapter == None:\n// error('Cannot pop last state {}.'.format(state.start))\n _pop();\n return true;\n }\n\n match = _beginChapterPattern.firstMatch(line);\n if (match != null) {\n var chapter = _book.findChapter(match[1]);\n var name = match[2];\n\n// if state.start != None:\n// old_chapter = book.chapter_number(state.start.chapter)\n// new_chapter = book.chapter_number(chapter)\n//\n// if chapter == state.start.chapter and name == state.start.name:\n// error('Pushing same snippet \"{} {}\"'.format(chapter, name))\n// if chapter == state.start.chapter:\n// error('Pushing same chapter, just use \"//>> {}\"'.format(name))\n// if new_chapter < old_chapter:\n// error('Can\\'t push earlier chapter \"{}\" from \"{}\".'.format(\n// chapter, state.start.chapter))\n _push(startChapter: chapter, startName: name);\n return true;\n }\n\n match = _endChapterPattern.firstMatch(line);\n if (match != null) {\n// var chapter = match[1];\n// var name = match[2];\n// if chapter != state.start.chapter or name != state.start.name:\n// error('Expecting to pop \"{} {}\" but got \"{} {}\".'.format(\n// state.start.chapter, state.start.name, chapter, name))\n// if state.start.chapter == None:\n// error('Cannot pop last state \"{}\".'.format(state.start))\n _pop();\n return true;\n }\n\n return false;\n }\n\n _ParseState get _currentState => _states.last;\n\n void _push(\n {Page startChapter, String startName, Page endChapter, String endName}) {\n startChapter ??= _currentState.start.chapter;\n\n CodeTag start;\n if (startName != null) {\n start = startChapter.findCodeTag(startName);\n } else {\n start = _currentState.start;\n }\n\n CodeTag end;\n if (endChapter != null) {\n end = endChapter.findCodeTag(endName);\n }\n\n _states.add(_ParseState(start, end));\n }\n\n void _pop() {\n _states.removeLast();\n }\n}\n\nclass _ParseState {\n final CodeTag start;\n final CodeTag end;\n\n _ParseState(this.start, [this.end]);\n\n String toString() {\n if (end != null) return \"_ParseState($start > $end)\";\n return \"_ParseState($start)\";\n }\n}\n" }, { "path": "tool/lib/src/split_chapter.dart", "content": "import 'dart:io';\n\nimport 'package:glob/glob.dart';\nimport 'package:path/path.dart' as p;\nimport 'package:pool/pool.dart';\n\nimport 'package:tool/src/book.dart';\nimport 'package:tool/src/code_tag.dart';\nimport 'package:tool/src/page.dart';\nimport 'package:tool/src/source_file_parser.dart';\n\n/// Don't do too many file operations at once or we risk running out of file\n/// descriptors.\nvar _filePool = Pool(200);\n\nFuture splitChapter(Book book, Page chapter, [CodeTag tag]) async {\n var futures = >[];\n\n for (var file in Glob(\"${chapter.language}/**.{c,h,java}\").listSync()) {\n futures.add(_splitSourceFile(book, chapter, file.path, tag));\n }\n\n await Future.wait(futures);\n}\n\nFuture _splitSourceFile(Book book, Page chapter, String sourcePath,\n [CodeTag tag]) async {\n var relative = p.relative(sourcePath, from: chapter.language);\n\n // Don't split the generated files.\n if (relative == \"com/craftinginterpreters/lox/Expr.java\") return;\n if (relative == \"com/craftinginterpreters/lox/Stmt.java\") return;\n\n var package = chapter.shortName;\n if (tag != null) {\n package = p.join(\"snippets\", package, tag.directory);\n }\n\n // If we're generating the split for an entire chapter, include all its\n // snippets.\n tag ??= book.lastSnippet(chapter).tag;\n\n var outputFile = File(p.join(\"gen\", package, relative));\n\n var resource = await _filePool.request();\n try {\n var output = _generateSourceFile(book, chapter, sourcePath, tag);\n if (output.isNotEmpty) {\n // Don't overwrite the file if it didn't change, so the makefile doesn't\n // think it was touched.\n if (await outputFile.exists()) {\n var previous = await outputFile.readAsString();\n if (previous == output) return;\n }\n\n // Write the changed output.\n await Directory(p.dirname(outputFile.path)).create(recursive: true);\n await outputFile.writeAsString(output);\n } else {\n // Remove it since it's supposed to be nonexistent.\n if (await outputFile.exists()) await outputFile.delete();\n }\n } finally {\n resource.release();\n }\n}\n\n/// Gets the code for [sourceFilePath] as it appears at [tag] of [chapter].\nString _generateSourceFile(\n Book book, Page chapter, String sourcePath, CodeTag tag) {\n var shortPath = p.relative(sourcePath, from: chapter.language);\n var sourceFile = SourceFileParser(book, sourcePath, shortPath).parse();\n\n var buffer = StringBuffer();\n for (var line in sourceFile.lines) {\n if (line.isPresent(tag)) {\n // Hack. In generate_ast.java, we split up a parameter list among\n // multiple chapters, which leads to hanging commas in some cases.\n // Remove them.\n if (line.text.trim().startsWith(\")\")) {\n var text = buffer.toString();\n if (text.endsWith(\",\\n\")) {\n buffer.clear();\n buffer.writeln(text.substring(0, text.length - 2));\n }\n }\n\n buffer.writeln(line.text);\n }\n }\n\n return buffer.toString();\n}\n" }, { "path": "tool/lib/src/syntax/grammar.dart", "content": "import 'language.dart';\nimport 'rule.dart';\n\nfinal languages = {\n \"c\": c,\n \"c++\": cpp,\n \"ebnf\": ebnf,\n \"java\": java,\n \"js\": js,\n \"lisp\": lisp,\n \"lox\": lox,\n // TODO: This is just enough for the one line in \"scanning\". Do more if\n // needed.\n \"lua\": Language(rules: _commonRules),\n \"python\": python,\n \"ruby\": ruby,\n};\n\nfinal c = Language(\n keywords: _cKeywords,\n types: \"bool char double FILE int size_t uint16_t uint32_t uint64_t uint8_t \"\n \"uintptr_t va_list void\",\n rules: _cRules,\n);\n\nfinal cpp = Language(\n keywords: _cKeywords,\n types: \"vector string\",\n rules: _cRules,\n);\n\nfinal ebnf = Language(\n rules: [\n // Color ALL_CAPS terminals like types to make them distinct.\n Rule(r\"[A-Z][A-Z0-9_]+\", \"t\"),\n ..._commonRules\n ],\n);\n\nfinal java = Language(\n keywords: \"abstract assert break case catch class const continue default do \"\n \"else enum extends false final finally for goto if implements import \"\n \"instanceof interface native new null package private protected public \"\n \"return static strictfp super switch synchronized this throw throws \"\n \"transient true try volatile while\",\n types: \"boolean byte char double float int long short void\",\n rules: [\n // Import.\n Rule.capture(r\"(import)(\\s+)(\\w+(?:\\.\\w+)*)(;)\", [\"k\", \"\", \"i\", \"\"]),\n // Static import.\n Rule.capture(r\"(import\\s+static?)(\\s+)(\\w+(?:\\.\\w+)*(?:\\.\\*)?)(;)\",\n [\"k\", \"\", \"i\", \"\"]),\n // Package.\n Rule.capture(r\"(package)(\\s+)(\\w+(?:\\.\\w+)*)(;)\", [\"k\", \"\", \"i\", \"\"]),\n // Annotation.\n Rule(r\"@[a-zA-Z_][a-zA-Z0-9_]*\", \"a\"),\n\n // ALL_CAPS constant names are colored like normal identifiers. We give\n // them their own rule so that it matches before the capitalized type name\n // rule.\n Rule(r\"[A-Z][A-Z0-9_]+\\b\", \"i\"),\n\n ..._commonRules,\n _characterRule,\n ],\n);\n\nfinal js = Language(\n keywords: \"break case catch class const continue debugger default delete do \"\n \"else export extends finally for function if import in instanceof let \"\n \"new return super switch this throw try typeof var void while with yield\",\n rules: _commonRules,\n);\n\nfinal lisp = Language(\n rules: [\n // TODO: Other punctuation characters.\n Rule(r\"[a-zA-Z0-9_-]+\", \"i\"),\n ],\n);\n\nfinal lox = Language(\n keywords: \"and class else false fun for if nil or print return super this \"\n \"true var while\",\n rules: _commonRules,\n);\n\nfinal python = Language(\n keywords: \"and as assert break class continue def del elif else except \"\n \"exec finally for from global if import in is lambda not or pass \"\n \"print raise range return try while with yield\",\n rules: _commonRules,\n);\n\nfinal ruby = Language(\n keywords: \"__LINE__ _ENCODING__ __FILE__ BEGIN END alias and begin break \"\n \"case class def defined? do else elsif end ensure false for if in lambda \"\n \"module next nil not or redo rescue retry return self super then true \"\n \"undef unless until when while yield\",\n rules: _commonRules,\n);\n\nfinal _cKeywords =\n \"break case const continue default do else enum extern false for goto if \"\n \"inline return sizeof static struct switch true typedef union while\";\n\nfinal _cRules = [\n // Preprocessor with comment.\n Rule.capture(r\"(#.*?)(//.*)\", [\"a\", \"c\"]),\n\n // Preprocessor.\n Rule(r\"#.*\", \"a\"),\n\n // ALL_CAPS preprocessor macro use.\n Rule(r\"[A-Z][A-Z0-9_]+\", \"a\"),\n\n ..._commonRules,\n _characterRule,\n];\n\n// TODO: Multi-character escapes?\nfinal _characterRule = Rule(r\"'\\\\?.'\", \"s\");\n\nfinal _commonRules = [\n StringRule(),\n\n Rule(r\"[0-9]+\\.[0-9]+f?\", \"n\"), // Float.\n Rule(r\"0x[0-9a-fA-F]+\", \"n\"), // Hex integer.\n Rule(r\"[0-9]+[Lu]?\", \"n\"), // Integer.\n\n Rule(r\"//.*\", \"c\"), // Line comment.\n\n // Capitalized type name.\n Rule(r\"[A-Z][A-Za-z0-9_]*\", \"t\"),\n\n // Other identifiers or keywords.\n IdentifierRule(),\n];\n" }, { "path": "tool/lib/src/syntax/highlighter.dart", "content": "import 'package:charcode/ascii.dart';\nimport 'package:string_scanner/string_scanner.dart';\n\nimport '../format.dart';\nimport '../term.dart' as term;\nimport 'grammar.dart' as grammar;\nimport 'language.dart';\n\nconst _maxLineLength = 67;\n\n/// Takes a string of source code and returns a block of HTML with spans for\n/// syntax highlighting.\n///\n/// Wraps the result in a 
     tag with the given [preClass].\nString formatCode(String language, List lines, Format format,\n    {String preClass, int indent = 0}) {\n  return Highlighter(language, format)._highlight(lines, preClass, indent);\n}\n\nvoid checkLineLength(String line) {\n  final asideCommentPattern = RegExp(r' +// \\[([-a-z0-9]+)\\]');\n  final asideWithCommentPattern = RegExp(r' +// (.+) \\[([-a-z0-9]+)\\]');\n\n  line = line.replaceAll(asideCommentPattern, '');\n  line = line.replaceAll(asideWithCommentPattern, '');\n\n  if (line.length <= _maxLineLength) return;\n\n  print(line.substring(0, _maxLineLength) +\n      term.red(line.substring(_maxLineLength)));\n}\n\nclass Highlighter {\n  final Format _format;\n  final StringBuffer _buffer = StringBuffer();\n  StringScanner scanner;\n  final Language language;\n\n  /// Whether we are in a multi-line macro started on a previous line.\n  bool _inMacro = false;\n\n  Highlighter(String language, this._format)\n      : language = grammar.languages[language] ??\n            (throw \"Unknown language '$language'.\");\n\n  String _highlight(List lines, String preClass, int indent) {\n    if (!_format.isPrint) {\n      _buffer.write(\"\");\n\n      // The HTML spec mandates that a leading newline after '
    ' is ignored.\n      // https://html.spec.whatwg.org/#element-restrictions\n      // Some snippets deliberately start with a newline which needs to be\n      // preserved, so output an extra (discarded) newline in that case.\n      if (_format.isWeb && lines.first.isEmpty) _buffer.writeln();\n    }\n\n    for (var line in lines) {\n      _scanLine(line, indent);\n    }\n\n    if (!_format.isPrint) _buffer.write(\"
    \");\n\n    return _buffer.toString();\n  }\n\n  void _scanLine(String line, int indent) {\n    if (line.trim().isEmpty) {\n      _buffer.writeln();\n      return;\n    }\n\n    // If the entire code block is indented, remove that indentation from the\n    // code lines.\n    if (line.length > indent) line = line.substring(indent);\n\n    checkLineLength(line);\n\n    // Hackish. If the line ends with `\\`, then it is a multi-line macro\n    // definition and we want to highlight subsequent lines like preprocessor\n    // code too.\n    if (language == grammar.c && line.endsWith(\"\\\\\")) _inMacro = true;\n\n    if (_inMacro) {\n      writeToken(\"a\", line);\n    } else {\n      scanner = StringScanner(line);\n      while (!scanner.isDone) {\n        var found = false;\n        for (var rule in language.rules) {\n          if (rule.apply(this)) {\n            found = true;\n            break;\n          }\n        }\n\n        if (!found) _writeChar(scanner.readChar());\n      }\n    }\n\n    if (_inMacro && !line.endsWith(\"\\\\\")) _inMacro = false;\n\n    _buffer.writeln();\n  }\n\n  void writeToken(String type, [String text]) {\n    text ??= scanner.lastMatch[0];\n\n    if (_format.isPrint) {\n      // Only highlight keywords and comments in XML.\n      var tag = {\"k\": \"keyword\", \"c\": \"comment\"}[type];\n\n      if (tag != null) _buffer.write(\"<$tag>\");\n      writeText(text);\n      if (tag != null) _buffer.write(\"\");\n    } else {\n      _buffer.write('');\n      writeText(text);\n      _buffer.write('');\n    }\n  }\n\n  void writeText(String string) {\n    for (var i = 0; i < string.length; i++) {\n      _writeChar(string.codeUnitAt(i));\n    }\n  }\n\n  void _writeChar(int char) {\n    switch (char) {\n      case $less_than:\n        _buffer.write(\"<\");\n        break;\n      case $greater_than:\n        _buffer.write(\">\");\n        break;\n      case $single_quote:\n        _buffer.write(\"'\");\n        break;\n      case $double_quote:\n        _buffer.write(\""\");\n        break;\n      case $ampersand:\n        _buffer.write(\"&\");\n        break;\n      default:\n        _buffer.writeCharCode(char);\n    }\n  }\n}\n"
  },
  {
    "path": "tool/lib/src/syntax/language.dart",
    "content": "import 'rule.dart';\n\n/// Defines the syntax rules for a single programming language.\nclass Language {\n  final Map words = {};\n  final List rules;\n\n  Language({String keywords, String types, List this.rules}) {\n    keywordType(String wordList, String type) {\n      if (wordList == null) return;\n      for (var word in wordList.split(\" \")) {\n        words[word] = type;\n      }\n    }\n\n    keywordType(keywords, \"k\");\n    keywordType(types, \"t\");\n  }\n}\n"
  },
  {
    "path": "tool/lib/src/syntax/rule.dart",
    "content": "import 'package:charcode/ascii.dart';\n\nimport 'highlighter.dart';\n\nabstract class Rule {\n  final RegExp pattern;\n\n  factory Rule(String pattern, String tokenType) =>\n      SimpleRule(pattern, tokenType);\n\n  factory Rule.capture(String pattern, List tokenTypes) =>\n      CaptureRule(pattern, tokenTypes);\n\n  Rule._(String pattern) : pattern = RegExp(pattern);\n\n  bool apply(Highlighter highlighter) {\n    if (!highlighter.scanner.scan(pattern)) return false;\n    applyRule(highlighter);\n    return true;\n  }\n\n  void applyRule(Highlighter highlighter);\n}\n\n/// Parses a single regex and outputs the entire matched text as a single token\n/// with the given [tokenType].\nclass SimpleRule extends Rule {\n  final String tokenType;\n\n  SimpleRule(String pattern, this.tokenType) : super._(pattern);\n\n  void applyRule(Highlighter highlighter) {\n    highlighter.writeToken(tokenType);\n  }\n}\n\n/// Parses a single regex where each capture group has a corresponding token\n/// type. If the type is `\"\"` for some group, the matched string text is output\n/// as plain text.\nclass CaptureRule extends Rule {\n  final List tokenTypes;\n\n  CaptureRule(String pattern, this.tokenTypes) : super._(pattern);\n\n  void applyRule(Highlighter highlighter) {\n    var match = highlighter.scanner.lastMatch;\n    for (var i = 0; i < tokenTypes.length; i++) {\n      var type = tokenTypes[i];\n      if (type.isNotEmpty) {\n        highlighter.writeToken(type, match[i + 1]);\n      } else {\n        highlighter.writeText(match[i + 1]);\n      }\n    }\n  }\n}\n\n/// Parses string literals and the escape codes inside them.\nclass StringRule extends Rule {\n  static final _escapePattern = RegExp(r\"\\\\.\");\n\n  StringRule() : super._('\"');\n\n  void applyRule(Highlighter highlighter) {\n    var scanner = highlighter.scanner;\n    var start = scanner.position - 1;\n\n    while (!scanner.isDone) {\n      if (scanner.scan(_escapePattern)) {\n        if (scanner.position > start) {\n          highlighter.writeToken(\n              \"s\", scanner.substring(start, scanner.position - 2));\n        }\n        highlighter.writeToken(\"e\");\n        start = scanner.position;\n      } else if (scanner.scanChar($double_quote)) {\n        highlighter.writeToken(\"s\", scanner.substring(start, scanner.position));\n        return;\n      } else {\n        scanner.position++;\n      }\n    }\n\n    // Error: Unterminated string.\n    highlighter.writeToken(\"err\", scanner.substring(start, scanner.position));\n  }\n}\n\n/// Parses an identifier and resolves keywords for their token type.\nclass IdentifierRule extends Rule {\n  IdentifierRule() : super._(r\"[a-zA-Z_][a-zA-Z0-9_]*\");\n\n  void applyRule(Highlighter highlighter) {\n    var identifier = highlighter.scanner.lastMatch[0];\n    var type = highlighter.language.words[identifier] ?? \"i\";\n    highlighter.writeToken(type);\n  }\n}\n"
  },
  {
    "path": "tool/lib/src/term.dart",
    "content": "/// Utilities for printing to the terminal.\nimport 'dart:io';\n\nfinal _cyan = _ansi('\\u001b[36m');\nfinal _gray = _ansi('\\u001b[1;30m');\nfinal _green = _ansi('\\u001b[32m');\nfinal _magenta = _ansi('\\u001b[35m');\nfinal _pink = _ansi('\\u001b[91m');\nfinal _red = _ansi('\\u001b[31m');\nfinal _yellow = _ansi('\\u001b[33m');\nfinal _none = _ansi('\\u001b[0m');\nfinal _resetColor = _ansi('\\u001b[39m');\n\nString cyan(Object message) => \"$_cyan$message$_none\";\nString gray(Object message) => \"$_gray$message$_none\";\nString green(Object message) => \"$_green$message$_resetColor\";\nString magenta(Object message) => \"$_magenta$message$_resetColor\";\nString pink(Object message) => \"$_pink$message$_resetColor\";\nString red(Object message) => \"$_red$message$_resetColor\";\nString yellow(Object message) => \"$_yellow$message$_resetColor\";\n\nvoid clearLine() {\n  if (_allowAnsi) {\n    stdout.write(\"\\u001b[2K\\r\");\n  } else {\n    print(\"\");\n  }\n}\n\nvoid writeLine([String line]) {\n  clearLine();\n  if (line != null) stdout.write(line);\n}\n\nbool get _allowAnsi =>\n    !Platform.isWindows && stdioType(stdout) == StdioType.terminal;\n\nString _ansi(String special, [String fallback = '']) =>\n    _allowAnsi ? special : fallback;\n"
  },
  {
    "path": "tool/lib/src/text.dart",
    "content": "import 'dart:convert';\nimport 'dart:math' as math;\n\n/// Punctuation characters removed from file names and anchors.\nfinal _punctuation = RegExp(r'[,.?!:' \"'\" '/\"()]');\n\nfinal _whitespace = RegExp(r\"\\s+\");\n\n/// Converts [text] to a string suitable for use as a file or anchor name.\nString toFileName(String text) {\n  if (text == \"Crafting Interpreters\") return \"index\";\n  if (text == \"Table of Contents\") return \"contents\";\n\n  // Hack. The introduction has a *subheader* named \"Challenges\" distinct from\n  // the challenges section. This function here is also used to generate the\n  // anchor names for the links, so handle that one specially so it doesn't\n  // collide with the real \"Challenges\" section.\n  if (text == \"Challenges\") return \"challenges_\";\n\n  return text.toLowerCase().replaceAll(\" \", \"-\").replaceAll(_punctuation, \"\");\n}\n\n/// Returns the length of the longest line in lines, or [longest], whichever\n/// is longer.\nint longestLine(int longest, Iterable lines) {\n  for (var line in lines) {\n    longest = math.max(longest, line.length);\n  }\n  return longest;\n}\n\nString pluralize(Iterable sequence) {\n  if (sequence.length == 1) return \"\";\n  return \"s\";\n}\n\nextension IntExtensions on int {\n  /// Convert n to roman numerals.\n  String get roman {\n    if (this <= 3) return \"I\" * this;\n    if (this == 4) return \"IV\";\n    if (this < 10) return \"V\" + \"I\" * (this - 5);\n\n    throw ArgumentError(\"Can't convert $this to Roman.\");\n  }\n\n  /// Make a nicely formatted string.\n  String get withCommas {\n    if (this > 1000) return \"${this ~/ 1000},${this % 1000}\";\n    return toString();\n  }\n}\n\nextension StringExtensions on String {\n  /// Use nicer HTML entities and special characters.\n  String get pretty {\n    return this\n        .replaceAll(\"à\", \"à\")\n        .replaceAll(\"ï\", \"ï\")\n        .replaceAll(\"ø\", \"ø\")\n        .replaceAll(\"æ\", \"æ\");\n  }\n\n  String get escapeHtml =>\n      const HtmlEscape(HtmlEscapeMode.attribute).convert(this);\n\n  int get wordCount => split(_whitespace).length;\n\n  /// Removes a single newline from the end of the string.\n  String trimTrailingNewline() {\n    if (endsWith(\"\\n\")) return substring(0, length - 1);\n    return this;\n  }\n}\n"
  },
  {
    "path": "tool/pubspec.yaml",
    "content": "name: tool\npublish_to: none\nenvironment:\n  sdk: '>2.11.0 <3.0.0'\ndependencies:\n  args: ^1.6.0\n  charcode: ^1.1.3\n  glob: ^1.2.0\n  image: ^2.1.19\n  markdown: ^2.1.3\n  mime_type: ^0.3.0\n  mustache_template: ^1.0.0\n  path: ^1.7.0\n  pool: ^1.4.0\n  sass: ^1.26.5\n  shelf: ^0.7.5\n  string_scanner: ^1.0.5\n"
  },
  {
    "path": "util/c.make",
    "content": "# Makefile for building a single configuration of the C interpreter. It expects\n# variables to be passed in for:\n#\n# MODE         \"debug\" or \"release\".\n# NAME         Name of the output executable (and object file directory).\n# SOURCE_DIR   Directory where source files and headers are found.\n\nifeq ($(CPP),true)\n\t# Ideally, we'd add -pedantic-errors, but the use of designated initializers\n\t# means clox relies on some GCC/Clang extensions to compile as C++.\n\tCFLAGS := -std=c++11\n\tC_LANG := -x c++\nelse\n\tCFLAGS := -std=c99\nendif\n\nCFLAGS += -Wall -Wextra -Werror -Wno-unused-parameter\n\n# If we're building at a point in the middle of a chapter, don't fail if there\n# are functions that aren't used yet.\nifeq ($(SNIPPET),true)\n\tCFLAGS += -Wno-unused-function\nendif\n\n# Mode configuration.\nifeq ($(MODE),debug)\n\tCFLAGS += -O0 -DDEBUG -g\n\tBUILD_DIR := build/debug\nelse\n\tCFLAGS += -O3 -flto\n\tBUILD_DIR := build/release\nendif\n\n# Files.\nHEADERS := $(wildcard $(SOURCE_DIR)/*.h)\nSOURCES := $(wildcard $(SOURCE_DIR)/*.c)\nOBJECTS := $(addprefix $(BUILD_DIR)/$(NAME)/, $(notdir $(SOURCES:.c=.o)))\n\n# Targets ---------------------------------------------------------------------\n\n# Link the interpreter.\nbuild/$(NAME): $(OBJECTS)\n\t@ printf \"%8s %-40s %s\\n\" $(CC) $@ \"$(CFLAGS)\"\n\t@ mkdir -p build\n\t@ $(CC) $(CFLAGS) $^ -o $@\n\n# Compile object files.\n$(BUILD_DIR)/$(NAME)/%.o: $(SOURCE_DIR)/%.c $(HEADERS)\n\t@ printf \"%8s %-40s %s\\n\" $(CC) $< \"$(CFLAGS)\"\n\t@ mkdir -p $(BUILD_DIR)/$(NAME)\n\t@ $(CC) -c $(C_LANG) $(CFLAGS) -o $@ $<\n\n.PHONY: default\n"
  },
  {
    "path": "util/intellij/chap04_read.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/chap05_scanning.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/chap06_representing.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/chap07_parsing.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/chap08_evaluating.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/chap09_statements.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/chap10_control.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/chap11_functions.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/chap12_resolving.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/chap13_classes.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/chap14_inheritance.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/intellij.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/jlox.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/intellij/section_test.iml",
    "content": "\n\n  \n    \n  \n"
  },
  {
    "path": "util/intellij/snippet_test.iml",
    "content": "\n\n  \n    \n    \n      \n    \n    \n    \n  \n"
  },
  {
    "path": "util/java.make",
    "content": "# Makefile for building a single directory of Java source files. It requires\n# a DIR variable to be set.\n\nBUILD_DIR := build\n\nSOURCES := $(wildcard $(DIR)/com/craftinginterpreters/$(PACKAGE)/*.java)\nCLASSES := $(addprefix $(BUILD_DIR)/, $(SOURCES:.java=.class))\n\nJAVA_OPTIONS := -Werror\n\ndefault: $(CLASSES)\n\t@: # Don't show \"Nothing to be done\" output.\n\n# Compile a single .java file to .class.\n$(BUILD_DIR)/$(DIR)/%.class: $(DIR)/%.java\n\t@ mkdir -p $(BUILD_DIR)/$(DIR)\n\t@ javac -cp $(DIR) -d $(BUILD_DIR)/$(DIR) $(JAVA_OPTIONS) -implicit:none $<\n\t@ printf \"%8s %-60s %s\\n\" javac $< \"$(JAVA_OPTIONS)\"\n\n.PHONY: default\n"
  }
]